Commit 7bad3db8 authored by Rémi Duraffort's avatar Rémi Duraffort

Merge branch 'http-cache' into 'master'

Allow to use KissCache automatically per dispatcher

See merge request !683
parents 653f5232 4d3a348c
Pipeline #6261 passed with stages
in 8 minutes and 10 seconds
.. _proxy:
Cache Proxy Setting Up
======================
HTTP proxies
============
Before, it used to use an internal cache mechanism for downloaded images and
hwpacks to avoid downloading repeatedly, which could save time and bandwidth.
When running jobs, LAVA is fetching many resources from remote servers
over http.
In some situation or when many jobs are running in parallel, the network
performances could become a bottleneck.
lava-dispatcher switches to use cache proxy for managing cache files
automatically. The recommended proxy is Squid.
To improve network performances, admins could setup a caching service that will
keep a local version of the resources used by LAVA.
Install Squid
^^^^^^^^^^^^^
Admin can choice among two kind of caching service:
Squid is easy to install via apt::
* generic http proxy like `squid <http://www.squid-cache.org>`_
* specific http cache like `KissCache <https://cache.lavasoftware.org/>`_
sudo apt install squid
Using the HTTP proxy
====================
Or if you want a configurable squid, refer to the following link to compile and
install manually: https://wiki.squid-cache.org/SquidFaq/CompilingSquid
The dispatcher will use the proxy configured in the HTTP_PROXY environment
variable.
Configure Squid
^^^^^^^^^^^^^^^
Environment variables are set in:
You will need to customize according to your server, like disk layout, space.
* ``/etc/lava-server/env.yaml`` for every dispatchers
* ``/etc/lava-server/dispatcher.d/<name>/env.yaml`` for a specific dispatcher
Need to analyze and tune by collecting information when squid running with real
cases, like cache policy, file system.
Using the HTTP cache
====================
Mandatory configuration options
-------------------------------
The dispatcher will use the caching service defined in the dispatcher
configuration in ``/etc/lava-server/dispatcher.d/<name>/dispatcher.yaml``.
Based on original /etc/squid/squid.conf, see below tuning.
Set ``http_url_format_string`` to the url of the local caching service.
* cache_dir ufs /var/spool/squid 30720 16 256
Mandatory option, please modify 30720(MB) to an available size.
There can be several cache directories on different disk, but it's better not
use RAID on the cache directories, it's recommended by Squid: The Definitive
Guide that it will always degrades fs performance for squid. 30720 is the
cache amount 30GB. 16 and 256 is Level 1 and 2 sub-directories, which is
default.
* maximum_object_size 1024000 KB
Mandatory option.
Setting the value as 1024000KB makes the squid cache large files less than
1GB, for our images are usually a large one but less than 1G.
Optional configuration options
------------------------------
Some others than mandatory options.
* acl over_conn_limit maxconn 10 # make max connection limit 10
* http_access allow localnet
Enable localnet, also, we need to define more localnet in server environment
to make sure all boards IP and other permitted clients are included.
acl localnet src 10.122.0.0/16
* http_access deny over_conn_limit
Make max connection of one client less than 10, it should be enough for
a board, it can be increased.
* cache_mem 128 MB
It can be increased if server MEM is enough, it's for squid mem amount for
objects.
* cache_swap_low 90
cache_swap_high 95
Cache size will maintain between 90% and 95%.
* client_lifetime 12 hours
Make a client continuous accessing time 12hrs, default is 1 days, it can be
increased.
* high_response_time_warning 2000
2s in 1mins no response will log in cache.log.
* There is some email configurations to be set, like 'cache_mgr', it will send
mail if cache proxy dies.
The configuration is only workable, there can be more improvement ways, some
still need to tune on server.
Other tuning
------------
Open files number can be increased for squid will need more than 1024
limitations sometimes::
# ulimit -n
1024
.. code-block:: yaml
http_url_format_string: "https://cache.lavasoftware.org/api/v1/fetch?url=%s"
.. robots:
......
......@@ -17,3 +17,9 @@
# /var/lib/lava/dispatcher/tmp/<prefix><job_id> instead of
# /var/lib/lava/dispatcher/tmp/<job_id>
# prefix: <prefix>
# Set this variable when using http caching service based on url substitution
# like KissCache
# When downloading resources, lava dispatcher will use this formating string
# instead of the original url.
# http_url_format_string: "https://cache.lavasoftware.org/api/v1/fetch/?url=%s"
......@@ -53,7 +53,7 @@ from lava_common.constants import (
from lava_dispatcher.actions.boot.fastboot import EnterFastbootAction
from lava_dispatcher.actions.boot.u_boot import UBootEnterFastbootAction
from urllib.parse import urlparse
from urllib.parse import quote_plus, urlparse
class DownloaderAction(RetryAction):
......@@ -127,6 +127,7 @@ class DownloadHandler(Action):
self.path = os.path.join(path, key) if uniquify else path
self.size = -1
self.decompress_command_map = {"xz": "unxz", "gz": "gunzip", "bz2": "bunzip2"}
self.fname = None
def reader(self):
raise LAVABug("'reader' function unimplemented")
......@@ -157,11 +158,11 @@ class DownloadHandler(Action):
self.url = urlparse(image["url"])
compression = image.get("compression")
archive = image.get("archive")
image_name, _ = self._url_to_fname_suffix(self.path, compression)
self.fname, _ = self._url_to_fname_suffix(self.path, compression)
image_arg = image.get("image_arg")
overlay = image.get("overlay", False)
self.set_namespace_data(
action="download-action", label=self.key, key="file", value=image_name
action="download-action", label=self.key, key="file", value=self.fname
)
self.set_namespace_data(
action="download-action",
......@@ -180,11 +181,11 @@ class DownloadHandler(Action):
compression = self.parameters[self.key].get("compression")
archive = self.parameters[self.key].get("archive")
overlay = self.parameters.get("overlay", False)
fname, _ = self._url_to_fname_suffix(self.path, compression)
if fname.endswith("/"):
self.fname, _ = self._url_to_fname_suffix(self.path, compression)
if self.fname.endswith("/"):
self.errors = "Cannot download a directory for %s" % self.key
self.set_namespace_data(
action="download-action", label=self.key, key="file", value=fname
action="download-action", label=self.key, key="file", value=self.fname
)
self.set_namespace_data(
action="download-action",
......@@ -270,14 +271,13 @@ class DownloadHandler(Action):
sha256sum = remote.get("sha256sum")
sha512sum = remote.get("sha512sum")
fname, _ = self._url_to_fname_suffix(self.path, compression)
if os.path.isdir(fname):
raise JobError("Download '%s' is a directory, not a file" % fname)
if os.path.exists(fname):
os.remove(fname)
if os.path.isdir(self.fname):
raise JobError("Download '%s' is a directory, not a file" % self.fname)
if os.path.exists(self.fname):
os.remove(self.fname)
self.logger.info("downloading %s", remote["url"])
self.logger.debug("saving as %s", fname)
self.logger.debug("saving as %s", self.fname)
downloaded_size = 0
beginning = time.time()
......@@ -321,12 +321,12 @@ class DownloadHandler(Action):
if compression and decompress_command:
try:
with open(fname, "wb") as dwnld_file:
with open(self.fname, "wb") as dwnld_file:
proc = subprocess.Popen( # nosec - internal.
[decompress_command], stdin=subprocess.PIPE, stdout=dwnld_file
)
except OSError as exc:
msg = "Unable to open %s: %s" % (fname, exc.strerror)
msg = "Unable to open %s: %s" % (self.fname, exc.strerror)
self.logger.error(msg)
raise InfrastructureError(msg)
......@@ -346,7 +346,7 @@ class DownloadHandler(Action):
raise JobError(error_message)
proc.wait()
else:
with open(fname, "wb") as dwnld_file:
with open(self.fname, "wb") as dwnld_file:
for buff in self.reader():
update_progress()
dwnld_file.write(buff)
......@@ -374,7 +374,7 @@ class DownloadHandler(Action):
# set the dynamic data into the context
self.set_namespace_data(
action="download-action", label=self.key, key="file", value=fname
action="download-action", label=self.key, key="file", value=self.fname
)
self.set_namespace_data(
action="download-action", label=self.key, key="md5", value=md5.hexdigest()
......@@ -398,7 +398,7 @@ class DownloadHandler(Action):
else:
archive = self.parameters[self.key].get("archive")
if archive:
origin = fname
origin = self.fname
target_fname = os.path.basename(origin).rstrip("." + archive)
target_fname_path = os.path.join(os.path.dirname(origin), target_fname)
if os.path.exists(target_fname_path):
......@@ -435,7 +435,7 @@ class DownloadHandler(Action):
)
)
self.results = {"fail": {"md5": md5sum, "download": chk_md5sum}}
raise JobError("MD5 checksum for '%s' does not match." % fname)
raise JobError("MD5 checksum for '%s' does not match." % self.fname)
self.results = {"success": {"md5": md5sum}}
if sha256sum is not None:
......@@ -455,7 +455,7 @@ class DownloadHandler(Action):
self.results = {
"fail": {"sha256": sha256sum, "download": chk_sha256sum}
}
raise JobError("SHA256 checksum for '%s' does not match." % fname)
raise JobError("SHA256 checksum for '%s' does not match." % self.fname)
self.results = {"success": {"sha256": sha256sum}}
if sha512sum is not None:
......@@ -475,7 +475,7 @@ class DownloadHandler(Action):
self.results = {
"fail": {"sha512": sha512sum, "download": chk_sha512sum}
}
raise JobError("SHA512 checksum for '%s' does not match." % fname)
raise JobError("SHA512 checksum for '%s' does not match." % self.fname)
self.results = {"success": {"sha512": sha512sum}}
# certain deployments need prefixes set
......@@ -487,7 +487,7 @@ class DownloadHandler(Action):
action="download-action",
label="file",
key=self.key,
value=os.path.join(suffix, self.key, os.path.basename(fname)),
value=os.path.join(suffix, self.key, os.path.basename(self.fname)),
)
elif self.parameters["to"] == "iso-installer":
suffix = self.get_namespace_data(
......@@ -497,11 +497,11 @@ class DownloadHandler(Action):
action="download-action",
label="file",
key=self.key,
value=os.path.join(suffix, self.key, os.path.basename(fname)),
value=os.path.join(suffix, self.key, os.path.basename(self.fname)),
)
else:
self.set_namespace_data(
action="download-action", label="file", key=self.key, value=fname
action="download-action", label="file", key=self.key, value=self.fname
)
# xnbd protocoll needs to know the location
......@@ -582,6 +582,18 @@ class HttpDownloadAction(DownloadHandler):
super().validate()
res = None
try:
http_cache = self.job.parameters["dispatcher"].get(
"http_url_format_string", ""
)
if http_cache:
self.logger.info("Using caching service: '%s'", http_cache)
try:
self.url = urlparse(http_cache % quote_plus(self.url.geturl()))
except TypeError as exc:
self.logger.error("Invalid http_url_format_string: '%s'", exc)
self.errors = "Invalid http_url_format_string: '%s'" % str(exc)
return
self.logger.debug("Validating that %s exists", self.url.geturl())
# Force the non-use of Accept-Encoding: gzip, this will permit to know the final size
res = requests.head(
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment