Syncing data via file protocols

In this section, we outline the protocols CitrusAd supports and how to name files so CitrusAd can download the data files automatically from your server.

Protocols

CitrusAd supports several ways to retrieve data files. Data files should be stored on your server, and files will be downloaded via one of the standard protocols.

Currently, CitrusAd supports the download of data files via the following protocols:

  • GCS
  • SFTP
  • SCP
  • HTTPS

In general, you will need to provide information about the protocol, host, port, and file path of a data file so CitrusAd can download it. When authentication is required to download data files, you will need to provide CitrusAd with a credential (e.g. username and password) to authenticate with your system.

If you are using an SFTP protocol, CitrusAd supports two types of authentication to download data files:

  • A username and a password
  • CitrusAd's public key.

In the second type of authentication, the below public key needs to be installed into your SFTP server.

Staging:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDMd4w5Z6hsjlUc0/r3dTJ+IaM9Mm+l3ZTrX3D5+30ZNiAb3Mhvlp8gv37CFKzqM90oFejl15bbr5dymjycsOWInxqkMyLOTbdMFAk7hmicNvN5bt3tx/Z0UewNvAx0Uw9Qxu7SCMSAalk9l7dqLp5RFFj+RHq9ApV8KIA/Fw2mjCw2vrJhiRWeuUuNNQTbGFBA5y72ATeYA2p0ZK3/TTJxyxCL70zGVxFG0jdAurfXKujYaF73Ql/gl5wmtnxGvTORtvObQdc2jCTsynj1w17LSZAdeoPQGnQkVCk37ZUnWTw8s4RYjHFVunsTDxJO5Qnw5yj8oqhe5IdpQyavUj+otRd6900q/3dK60IELH6AFz+ZEqudlBh8ZO2hvHIDS2Ii5S3LunlejykAVdRl7RVHA17Pk3mmrHCS6i3n9ZliBYeVQiKme2jHVB+ESrHFxQWl3ct9MYFNlAqwuhTFQRSG9ioSEHX5Dnmqk8aqbm/4pYY4b7YHOcIpHrtrVrFTJ3vLM1If1OVrDQidFhi1ArmoCU+PO65jNSeMj27m/HoX7Mbcmi8F++Op3MgP8iJrohE0Wjz+use7nlDtQTtDxDeEQQYGqsz4DOtH/jn1TWRynaXZqwPqC5Q6b52QM22T+E3rgT/4G7yJU3IRKioByOck251MEtOEgSCvvIAaV685fQ==

Production

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCtpIUFoYu8OUesZGP9rffv6/hky1HbQCAhC8hNRmcWpdptzRYsKIZFPz9t2vb85dierTZqJstAEJkePHd25cZ8btIBacbAR49avCIXiBuVgvlJk1+Er2dyYqSzSeFhpzlZxi4CkGo/Nsx7yxPi5693fdVF3vr+/iencw1tdLJJXfT6j1YLBCREg+IziBCO6m9P0QQGEDXYvsOmIzl0CJ8enU7gjvnYL5ify1/aGpoXv9/XcCYJWXTQmvmISknOMdC4pBiuDqsE9CX27FLQQb1ZT5jNrqD5hFsEo0rWqLiCbb5qoxIdMSe/IbA+sewgpgTPNJXeCNLSSH/MHhj2IgiMZGDDDaVaqMZD7N2jeQdI6SQbgvXiLTTUL4gNyO5eN9oqoeJumeZMi0+MQ7kkfHDYGtKe3RMxL7heL650HL+eWDIybezyPdQmfyVew9KuN0uB+2X+2NxQs8ZLqow486XDfUDfo+rPuoIvkYy7Z8ox+gne+1iAk43TCekcDwxlc56R9muMtGLuzzg1x9s9MqsAg5JNWEPQ8giAwCQJwm7h3UcCY3TJCn0LUzYHc7Ycr11JfIXL47vYGH3C0XBYlbEU/8m4VhhexzgJi8G/reclY5pfjtwQGOiobsxLsWgGx6Qoz/qWqmGwQIg+qi7jvA7jnltUP+RTFZpmd18/dSxB+w==

Production:

Data compression and encryption

You may need to compress and encrypt data before uploading it to your server for syncing. When both compression and encryption are used on data files, CitrusAd assumes that the data files are compressed prior to encryption.

When a data file is encrypted, CitrusAd will decrypt the data file after it’s been downloaded. Currently, we support data decryption for files that are encrypted by PGP programs. More information about PGP cryptography can be found at https://tools.ietf.org/html/rfc4880.

If you choose to use this type of cryptography for data files, you can use the below public key sto encrypt data files before uploading the files to your server. Only CitrusAd can decrypt the encrypted data files.

Staging:

-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBFxJU8wBEADYN17whE2HZ0HGVms8UBolerNfLpTzdlH2TxzdJB9RXC5XxO7K
x0dOTDncBLF5IuJ9vRWF802xa9vga6tId83Hf9Qf1ONSWRGHsX1KNtQX7liSQ0Sh
BWdNPlIobYWWkid6s4Cbyk8K/GPvgDdY5adwSMLl2Zj+mkP+b9TBnR8/WfpeYwIn
Yrmurxe6RLZZ7LqoYuxPyCMyIMUpqS8HCdyz4yAgrOPHQNi2K5jbsv9jAH6BSb+z
Uq6BJvNdyjV6y2QR+qHy55iVj1DNh5RDsRX/fa8yCLz6JHhzmMemMW4OKwvYSkor
Vo7njJvCrqIm4QbmHMZ3kFGBCNdiQ8OK/dYolBOS4J4a07d06DKIvLB9hH7k1L3i
GLvz0WrUPrL8D6Q+hDoUoEY/2LFL2F7eZdpDLl6lE1et9sV8Ooz9nMgjRpTSaMiR
p/Jzwtp6gkuMxvz1vQvhEMFI5osD5lAVjmPyzjOsF0awDkehmROcxDeazHLZZy2/
sj8rkllzBaCFQzDIKak27+sVGmeoxLApN36y6wo/6CM3MoJFrJWVE5ERhD5lPBgw
uey2WNPPtOmLoXhkDPyJKwKmLaS26NkxBS/Z7OuL2iu3x9Qqi8pGaLicFSP8tvOg
+jh1MufT0iMR4g4zAFw22+wELOTKW3kbw5llxCVkfb+9wqTcm95XeqiccwARAQAB
tFBDaXRydXMgSW50ZXJuYXRpb25hbCAoUEdQIGtleSBmb3IgQ29sZXMgU0ZUUCBh
dXRoZW50aWNhdGlvbikgPGluZm9AY2l0cnVzYWQuY29tPokCTgQTAQgAOBYhBCfj
FA8PU/poz61yfYp13Hlao4gZBQJcSVPMAhsDBQsJCAcCBhUKCQgLAgQWAgMBAh4B
AheAAAoJEIp13Hlao4gZsVIQAMMilxZMT7jgheZO0zqZNRgExkGzzjs0/58vJKyl
/U31iMVWR/lWXWgjqwWAhL9sV9OtI6CKBRJEq+TnLG88726uk0K5gdkvntmdJY4A
+ypCMuNoccph7HCNJiFFrCaEAhYYp3i8XNBjUJCC/re8eA/oX0iXIojIkUc6CUzf
q4+De9Uozv7Uqi9sGlhjnscF4KyboA/K1yEoXVWoc/5DOqLUz3nIeWVA9pz0h9tB
iGoYW3tWQndsXxiFCGK9jbOwkuj5NuWNgYDQETiB7cAjqBeAKbWG9uPZ6i7yI6Yr
lDLiaWf326s0eDgS+IaCNV+w0vBvMpwcuuCo2Qn7OAUpU5p7glV1FOrzcHBOcKzp
LAjzjVaxt/+eIbJyA1Xg+K4enJzDLJlDLBGA1B+RQtuXiufPfzXbwagQM0l+8Rd+
QD8L2Dlirz6mUfhqz5Pytinb0fnP6MIcxVp7p/cHDk/ICcSMVi16tlYOuN6ENUor
gDOekBaCFm7r1T5+UYA8y3ma7eFdD39u3qneWqreI8mvaet1o59LR65uFiXh5nZt
9MTrnT26Pg2AH6IPwqDjkOWbpnV7wiXuq71hudV1kYqIJF9ELbHOfxVHran1aAYQ
WdRllxU+yr0DTSbo1xMGtQsLlDqwhj/M5wGBbq5MC4Xo/J3vHUPysiIXFZqRKIRd
1HAUuQINBFxJU8wBEADQKHy7btTlmXnwQehUnD9a1PpjRYG/YecHS/k0joUElBgu
raQUVo2X2m+CBuqndI0s4sxvhTTVffxWuO2wjpUPN2XBiekNZjVJei1Ky5ieuVwF
WmA4iZLtU/PkQzQl/L70mkoe5H+kgLkrw9PY98Z7dy+fh02nVgg2juIy0imkzoqC
Ts5EIfppfBdDwabLG4WTl4SwFp7vV0N50xjJVBEWeu+rdshz9zQYLb9USbOpslDG
AZfPV1wFBv3sxvRMDODGBW3FcczzkoPqZK8UO6SdYPYx8aICR1d5W5WHjHU6qJ6d
F2KKV2uyD5WWMElhgtMo4ZkOEr8ogLjrwFfsSR4kTIsOCx4bN604oJ5Hv4uoCTlZ
cIDlW8RtgBw4pAjFbXo1+PkLG/6v/0PhG/MSgUPoKJopL2cPE+1R4UFskpj/+7DK
fT27oRb7qvWGmNH5D8Vb91fN4F3zHIT6bbn73PsQ9rDUkMgZzv3iaZNPOAeg6UEf
GTt+kNCJTmxp6ePwGxjqcN5wXcMkS7ng4mROrCuWMzDduqEPuXhPjyNZL7KxME3N
IhRFaa+8yf7LIkzupEH986o7+7gMoiNR/M/IJH8glj3EvihYvRecJC4zPRf+PzPB
bUQlayf0VQaayrT05nrEsV5FVaZR1STN3HUaEZBseQJFy2eknwRPXIp7QDeeEwAR
AQABiQI2BBgBCAAgFiEEJ+MUDw9T+mjPrXJ9inXceVqjiBkFAlxJU8wCGwwACgkQ
inXceVqjiBkH4A//awlAszaDm1yX4gn+kekzC/r8kaiv7qIkvTK2wVzYEUt3X64d
FbhgWZtVBqDjhWWP4ghV6+7iZfYbFu1c0e51dIpECaYByqnJN0+vOJZzo6nGh4/S
aucdF/eHW2ghJ7VswK8aAwMYCVM8g2PSei6DnoXAdL6UjfkZKwaeFD8/IPLBmahE
yGSBNm/2zRymwR3QnEpiBQ1EBGinU73KS+Ssl2wYypUJaRwXBBeAO1uYZYtRpeyb
ERA3IRsP1KCoj+a/GgLa7Uwu23V8GwA76TIkjXko0sDzGe7jycMYozgsUH94gpoN
VCyPyg45m9j3rjD9XPV0dJsrf3M5H3WdR6os6geE5fsNB+b3rCM6npgJWw7UHlMj
UV7ckCew3rZt/JIX77o/yPtcZPd4ssMkj4miPBo2YXErkkB3L22JQN2/MG/b5Vzy
84M5O9q8PYKenjmRFz4HZefB1N8lUeL0c01iHI/0N1htGEIi/RmZ1SuOICQicWaZ
v3sQFW4gGmsjq97gnWw2+An4uGhM3YOrTtq7dVv7bToC+JWnN1t0PUJ88aNMzEhF
0IPAWKJpioKwtXbF/xhLvwUy4XRfMY8/XXY/q08kjN5BhHP1X8ZTcquKEI+5LQrJ
hloK/WTQUotAPWXZixOAwyILDgez+a4dTqb91bf0sho5uB/Wk+vxoqnCvBU=
=IBS4
-----END PGP PUBLIC KEY BLOCK-----

Production:

-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBFxmGOgBEAC1nFt5/7zo8Q3AaweOq4JlDzPxnil8f/EYnYccsuKLkqmAthC4
TvGssIHJ/NBUUBb07ylkBrdcQhHLwPhemqJekaEI3vSDA1+9DbxjbnSHp6BAkyhn
whWmDGttknun7chekKhR+ijH8qCz9xMIZNugdt5huJyevGcFduKH22Ad+4kU4xQf
FQBR/z++1cVDCWFQKloGX61faUKMr2eBvP2L3e71U6HVKuthVYZerzYFAXvl1LCW
UQ86fJIG9qOgvIEgPcHj8fpCE8QQ5z8RG4aPK6u40oPZUJ3uZSokJAjD/EZONKSG
8u6oX3BWVoABSS2iVhRz541nUGVdDX4xLl0vi4Lq0FXtk4t+SXXqiTENT6W/XL2o
6mfUqeMAPG0+TL1Pbk5Zq+xaeuWcXZ9hv58129XR2MXWHJz7d5EL0VmUmYOMIZWw
uzmRsQrVT+4QmYOFgPoa2U0fhmsNTXLw+oFt7tySoGtfC512D69GNp4GQzj8LSCL
P6Oxml4eueM/A0i8m7vOI2J1gfQArd/tFYYGg6oZ+heOgBvwrtCUd2QiWdg+jjHL
PA1avpnt517YvBi3smUUT90blBqjsaJIcDaehMjraJ4otvKegOtnbKUWyZuxNYun
zM1PSCKLLFojbhd3nMoqF+ZQP27bBCzYJc3zk7dzdwWiE7OX/ddBDj36WQARAQAB
tFZDaXRydXMgSW50ZXJuYXRpb25hbCAoUEdQIGtleSBmb3IgY2F0YWxvZyBlbmNy
eXB0aW9uIC0gUHJvZHVjdGlvbikgPGluZm9AY2l0cnVzYWQuY29tPokCTgQTAQgA
OBYhBOgn+S5kYTP5OK9o9ZD+RIlYHzF5BQJcZhjoAhsDBQsJCAcCBhUKCQgLAgQW
AgMBAh4BAheAAAoJEJD+RIlYHzF5Mh8P/3XoQ9yf6eTR+y52LiBuF9dUKpxPkYE5
b/6PUuI4dl5znyxNXF6YaI50IewuQG9+55atEqovCJlSfZRQ7xH1W8VxwOSiisTJ
iuwWxXwyDFq7JnnHHLrPswJj2wOLTNjZpnXBHH7mh/AX2rEXueTvfzMYQaaltOJb
PTnbEVrBWZ11BK2JJCnmvDdEzm4i0b/poxm4wvaBFHpNhLmagPrAnJVQGgKhPY+X
gpr3R7jQAm8BdL53v3aG4kZOQxkrB1j5yLmZK8xQkjURAZs8Ps8lYp4NLN/OLE0e
9hf0skRWwDdOD4kxFmLTBW73Wtj/+QCTbyVtCrm+YVmDQ9nw+WEpdMZ4HMtBBs44
hyl0rfMAqpMaX04LTA6kilGg6+govm5W31FsOsQpYy002j4HpvKXWEmMfipuu4db
Y7ep1thyU+UVjk4OQLK3EArdT6KRGGO9QA5vXkA3EkP8ItBIGAouR62nckspNDQo
ryiEDt/aFqEAbslFhD6k3SmNAe9cM+/Vx5m2w3UPThzVekpZZnxTo6dFr4vt6DWQ
OjBSPa2jbKWh9an78FRwS+tToixVV1QHllfw9kWTdfHXMfozot03ZvzCcLq4asvH
5zQB1S9hIQXtJqHAM2f+3MgGYXQpn0pn5x8Dd8u5n63gjJE1cr+xau6BEfI/a67L
jIpSRDDhltIHuQINBFxmGOgBEADbrx3iclfeSoaE/cdEfpqFizvRvgpv/bi4CQKn
fX4uOurVYInD1H5/XdVYjfdc9RHhXkeIdieFh7GI3tnrGKNGUZ+7zVg7wtzwnX75
gUgTFLBJSriyuZW+GnRU5xHMUAd/jhp2tLGlsgmBuUGy6bMf9sWkz8dUsaW1u3DQ
GjAAMGdoYzBJCWLd5VWMWfxbQ8hSfjMguVs19l06LDiu4OkL2euAFprFTVhK/Zt4
wa/sRMpiLLgoJvcxfIHS+CmdeY5mm5QcaKj8JBnQIWZt88XlrtZY6DxLRu3SrX1A
10RBOqwyjdU25RV93KzBhPDnPdAVAk12wIHuArxuYJTuGrgPESRBNYVlp7hQmR3D
a7S37YkMJDu5PcR9gn4J1L4GrzdeDtvIkNVz+/XSpeFsmrjcsE5IlFKYibHkJoWk
AoRpkW/COsIQRGV1CySvMmVHnrs1RMMhbRcE1YMyFmedfcmyuuIDdNMOM9UdwyBj
I1Oz3iebIQ/aXgohVCuCM0k8o3wFa11Ci9Ag2svoA5/tatPCb770t0tOC05sTHlJ
xwx5Jpnd0yObjGIm63u0wF4lCznzxOdx/mq/9H2y1/vKl18+aHYwwOz7AGGcv5l9
oMJR7o3oimZ/qTCSldsjBWaRHebk3jIw0dkxvP3vq+TT3/URkcudUnCv+AnXfVRV
rgxtKQARAQABiQI2BBgBCAAgFiEE6Cf5LmRhM/k4r2j1kP5EiVgfMXkFAlxmGOgC
GwwACgkQkP5EiVgfMXmygA/+Mnd6i1IW14To6TJUP3i8aG/+RJiqREzDao8YX9ck
bFzsz3XY1kbmXAfcQReWpfk/OSI9y2LivfsFR0AZiZAUqrf7mqp7W7SbFRB/01uG
TZW3OjQ5m+UUez7I4aYXaIXWryfRMoSCPkT6UBpNdxTStuUAkR8SJtqGQy6hOAgy
h6ER9M0Hf0VDbATzEJdrwsTSjBjp/o29mAFQUWwRl9IntpHc2ouu1SYmRyIjcyca
Nkb3yp59HgbtNjbs1cHH++/IJ/6+cnfzlQpGFnv++SWRtNz9wZPfLHQZe6uSo59r
z3+yeHX70aDFYxeNuPUkILein9h4MSrcsKtAkRReMdx5Bq8inXYI92edGaJaR661
yb22rbbA7o0+XFls1oHavecjyLyBPsmb5qJ6zoBz8PuZ1uivXQeilMpxW434B8MY
MIOXvAOPcrNiKfi57C61x6nfoepCcoU9xg5/xYp2mny0NJV0PUrOUz7dja/ZXLAv
VOKgW7cyEwZ9K2WnIq5PBZ3GZgFsPNa5EkxdFTfALZS+kuqbdg8yNaY+xyBn3AUs
waYUruHhygKEgOBhHZBWobNxile+nfOd/7cn5PmP4aONdlNzUXfBko25ZfxsUW5l
yzi1pUho07hvO/AEDInUWEzATJJIS+9WwoM0tl86PmeQ/R8DZNcAEmPBFVfd56Ad
tPA=
=lRkN
-----END PGP PUBLIC KEY BLOCK-----

When a data file is delivered compressed, we will decompress the data file prior to processing. Currently, we support decompression of two types of compression formats: zip and gzip.

File naming convention

As mentioned previously, you will need to provide information about the protocol, host, port, and file path to a data file so CitrusAd can download and process the data file. You can choose a daily time that’s convenient to ensure that the data file is ready on the server each day, and CitrusAd will download the data files daily.

The simplest way to specify a target file for CitrusAd to download each day is through providing an explicit file name. CitrusAd will then use this file name to retrieve the data file from your server.

When you use FTP, FTPS, and SFTP protocols for communication between your server and CitrusAd, we support other options for you to specify target files for CitrusAd to download, including:

  • Rolling_earliest
  • Rolling_earliest_24_hours
  • Rolling_latest
  • Rolling_latest_24_hours.

They are also called target file modes.

When you choose one of the options outlined above, you will need to provide CitrusAd with a textual template for the data file names. In the textual template, there is a special string, which is “{*}”. CitrusAd will use the template provided by you to match the file names on your server to choose and download a target file each day.

An example of a template can be “CitrusAdCatalogDataAU{}.txt”. The template defines that the matched file names must start with the prefix “CitrusAdCatalogData_AU” and end with the suffix “.txt”. When the template “CitrusAdCatalogDataAU{_}.txt” is used, the below file names will match with the template:

CitrusAdCatalogProduct_AU_20190315.txt
CitrusAdCatalogProduct_AU_20190314.txt
CitrusAdCatalogProduct_AU_20190312.txt

In order to avoid downloading data files that you are uploading, CitrusAd only downloads data files that have been modified more than one minute from the time we access the server.

Although there are several file names that match the template, CitrusAd only downloads and processes one file at a time. In order to choose a file from a list of candidates, we define different target file modes you can choose from. Target file modes are discussed in more detail below.

Rolling_earliest

In this target file mode, we use the template to filter files by using their names, then we sort the results by ascending file names and return the first result.

For example, if the template for file names is “CitrusAdCatalogDataAU{*}.txt” and the list of file names filtered by the template are as below, the file “CitrusAdCatalogProduct_AU_20190312.txt” will be chosen to download in this target file mode.

CitrusAdCatalogProduct_AU_20190312.txt
CitrusAdCatalogProduct_AU_20190313.txt
CitrusAdCatalogProduct_AU_20190314.txt

Rolling_earliest_24_hours

In this target file mode, we first use the template to filter files using their names. Then we only choose files that are modified within the recent twenty-four hours. Finally, we sort the results by ascending file names and return the first result.

For example, we assume that the current time is 2019-03-15 10:30:07 and the template for file names is “CitrusAdCatalogDataAU{*}.txt”. If the list of file names that are filtered by the template is in the table below, the file “CitrusAdCatalogProduct_AU_20190314.txt” will be chosen to download in this target file mode.

An example of files with names and last modified information

File nameLast modified
CitrusAdCatalogProduct_AU_20190312.txt13/03/19 15:35:11
CitrusAdCatalogProduct_AU_20190313.txt13/03/19 15:35:08
CitrusAdCatalogProduct_AU_20190314.txt14/03/19 15:35:10
CitrusAdCatalogProduct_AU_20190315.txt15/03/19 10:05:07

Rolling_latest

In this target file mode, we use the template to filter files using their names. Then we sort the results by descending file names and return the first result.

For example, if the template for file names is “CitrusAdCatalogDataAU{*}.txt” and the list of file names filtered by the template is as below, the file “CitrusAdCatalogProduct_AU_20190314.txt” will be chosen to download in this target file mode.

CitrusAdCatalogProduct_AU_20190314.txt
CitrusAdCatalogProduct_AU_20190313.txt
CitrusAdCatalogProduct_AU_20190312.txt

This target file mode is similar to Rolling_earliest. However, instead of sorting the files by ascending file names, we sort them by descending file names.

Rolling_latest_24_hours

In this target file mode, we first use the template to filter files by using their names. Then we only choose files which are modified within the recent twenty four hours. Finally, we sort the results by descending file names and return the first result.

For example, we assume that the current time is 2019-03-15 10:30:07 and the template for file names is “CitrusAdCatalogDataAU{*}.txt”. If the list of file names that are filtered by the template is in the table below, the file “CitrusAdCatalogProduct_AU_20190315.txt” will be chosen to download in this target file mode.

This target file mode is similar to Rolling_earliest_24_hours. However, instead of sorting the files by ascending file names, we sort them by descending file names.

An example of files with names and last modified information

File nameLast modified
CitrusAdCatalogProduct_AU_20190312.txt13/03/19 15:35:11
CitrusAdCatalogProduct_AU_20190313.txt13/03/19 15:35:08
CitrusAdCatalogProduct_AU_20190314.txt14/03/19 15:35:10
CitrusAdCatalogProduct_AU_20190315.txt15/03/19 10:05:07