1
0
Fork 0
mirror of https://github.com/moby/moby.git synced 2022-11-09 12:21:53 -05:00
Commit graph

25 commits

Author SHA1 Message Date
Yong Tang
9528ea930c Sanitize docker labels when used as journald field names
This fix tries to address the issue raised in #23528 where
docker labels caused journald log error because journald
has special requirements on field names.

This fix addresses this issue by sanitize the labels per
requirements of journald.

Additional unit tests have been added to cover the changes.

This fix fixes #23528.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
2016-08-05 15:20:47 -07:00
Sebastiaan van Stijn
48569e0a35 Merge pull request #22911 from vdemeester/20033-default-logging-tag-value
Standardize default logging tag value
2016-07-29 02:18:57 +02:00
Nalin Dahyabhai
513ec73831 Improve logging of long log lines
This change updates how we handle long lines of output from the
container.  The previous logic used a bufio reader to read entire lines
of output from the container through an intermediate BytesPipe, and that
allowed the container to cause dockerd to consume an unconstrained
amount of memory as it attempted to collect a whole line of output, by
outputting data without newlines.

To avoid that, we replace the bufio reader with our own buffering scheme
that handles log lines up to 16k in length, breaking up anything longer
than that into multiple chunks.  If we can dispense with noting this
detail properly at the end of output, we can switch from using
ReadBytes() to using ReadLine() instead.  We add a field ("Partial") to
the log message structure to flag when we pass data to the log driver
that did not end with a newline.

The Line member of Message structures that we pass to log drivers is now
a slice into data which can be overwritten between calls to the log
driver's Log() method, so drivers which batch up Messages before
processing them need to take additional care: we add a function
(logger.CopyMessage()) that can be used to create a deep copy of a
Message structure, and modify the awslogs driver to use it.

We update the jsonfile log driver to append a "\n" to the data that it
logs to disk only when the Partial flag is false (it previously did so
unconditionally), to make its "logs" output correctly reproduce the data
as we received it.

Likewise, we modify the journald log driver to add a data field with
value CONTAINER_PARTIAL_MESSAGE=true to entries when the Partial flag is
true, and update its "logs" reader to refrain from appending a "\n" to
the data that it retrieves if it does not see this field/value pair (it
also previously did this unconditionally).

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2016-06-14 14:11:47 -04:00
Vincent Demeester
f900e1cf47
Standardize default logging tag value
Use the same default tag value for all loggers that support tags.

Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2016-06-04 12:38:12 +02:00
Nalin Dahyabhai
0da0a8f9da Add support for reading journal extras and in UTC
When told to read additional attributes from logs that we've sent to the
journal, pull out all of the non-trusted, non-user fields that we didn't
hard-code ourselves.  More of PR#20726 and PR#21889.

When reading entries in the journald log reader, set the time zone on
timestamps that we read to UTC, so that we send UTC values to the client
instead of values that are local to whatever timezone dockerd happens to
be running in.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2016-06-02 10:17:07 -04:00
Nalin Dahyabhai
7772d270c0 Remove the logger.Message ContainerID field
Log drivers are instantiated on a per-container basis, and passed the
container ID (along with other information) when they're initialized.
Drivers that care about that value are caching the value that they're
passed when they're initialized and using it in favor of the value
contained in Message structures that are passed to them, so the field in
Messages is unused, so we remove it.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2016-05-31 16:41:29 -04:00
Nalin Dahyabhai
ab62ecf393 Open the journald following descriptor earlier
Following a journal log almost always requires a descriptor to be
allocated.  In cases where we're running out of descriptors, this means
we might get stuck while attempting to start following the journal, at a
point where it's too late to report it to the client and clean up
easily.  The journal reading context will cache the value once it's
allocated, so here we move the check earlier, so that we can detect a
problem when we can still report it cleanly.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
2016-03-24 10:12:51 -04:00
Nalin Dahyabhai
8d597d25a8 Improve error reporting when following journals
When we set up to start following a journal, if we get error results
from sd_journal_get_fd() or sd_journal_get_events() that prevent us from
following the journal, report the error instead of just mysteriously
failing.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2016-03-24 10:12:15 -04:00
Nalin Dahyabhai
52c0f36f7b Fix a race in cleaning up after journald followers
When following a journal-based log, it was possible for the worker
goroutine, which reads the journal using the journal context and sends
entry data down the message channel, to be scheduled after the function
which started it had returned.  This could create problems, since the
invoking function was closing the journal context object and message
channel before it returned, which could trigger use-after-free segfaults
and write-to-closed-channel panics in the worker goroutine.

Make the cleanup in the invoking function conditional so that it's only
done when we're not following the logs, and if we are, that it's left to
the worker goroutine to close them.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2016-03-17 18:36:21 -04:00
Nalin Dahyabhai
4d200cd693 Fix a race in maintaining the journald reader list
The journald log reader keeps a map of following readers so that it can
close them properly when the journald reader object itself is closed,
but it was possible for its worker goroutine to be scheduled so that the
worker attempted to remove a reader from the map before the reader had
been added to the map.  This patch adds the item to the map before
starting the goroutine which is expected to eventually remove it.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2016-03-17 18:36:21 -04:00
Nalin Dahyabhai
6cdc4ba6cd Try to handle changing names for journal packages
When checking if we have the development files for libsystemd's journal
APIs, check for either 'libsystemd >= 209' and 'libsystemd-journal'.  If
we find 'libsystemd', define the 'journald' tag, which defaults to using
the 'libsystemd.pc' file.  If we find the older 'libsystemd-journal',
define both the 'journald' and 'journald_compat' tags, which causes the
'libsystemd-journal.pc' file to be consulted instead.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2016-02-23 12:24:27 -05:00
Ivan Babrou
5a3351883b Add tag support to journald logging driver, closes #19556
Signed-off-by: Ivan Babrou <ibobrik@gmail.com>
2016-01-27 10:52:19 +00:00
Nalin Dahyabhai
0ca6d77e6e Revert "prevent journald from being built on ARM"
This reverts commit 6f6f10a75f, so that we
can apply a different workaround.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2015-12-02 10:30:13 -05:00
Stefan Scherer
6f6f10a75f prevent journald from being built on ARM
Signed-off-by: Govinda Fichtner <govinda.fichtner@googlemail.com>
2015-11-21 15:17:31 +01:00
John Howard
5452954d89 Windows: Fix journald compile error
Signed-off-by: John Howard <jhoward@microsoft.com>
2015-10-31 08:31:25 -07:00
Daniel Dao
11a24f19c2 add labels/env log option for journald
this allows journald logger to collect extra metadata from containers with
`--log-opt labels=label1,label2 --log-opt env=env1,env2`

Signed-off-by: Daniel Dao <dqminh@cloudflare.com>
2015-10-12 21:12:46 +02:00
Nalin Dahyabhai
11fda783f8 Remove unnecessary check for nil CString
@noxiouz points out that we don't need to check for a nil result from
C.CString(), since an out-of-memory condition causes a runtime panic
instead.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2015-09-14 14:16:48 -04:00
Nalin Dahyabhai
e611a189cb Add log reading to the journald log driver
If a logdriver doesn't register a callback function to validate log
options, it won't be usable.  Fix the journald driver by adding a dummy
validator.

Teach the client and the daemon's "logs" logic that the server can also
supply "logs" data via the "journald" driver.  Update documentation and
tests that depend on error messages.

Add support for reading log data from the systemd journal to the
journald log driver.  The internal logic uses a goroutine to scan the
journal for matching entries after any specified cutoff time, formats
the messages from those entries as JSONLog messages, and stuffs the
results down a pipe whose reading end we hand back to the caller.

If we are missing any of the 'linux', 'cgo', or 'journald' build tags,
however, we don't implement a reader, so the 'logs' endpoint will still
return an error.

Make the necessary changes to the build setup to ensure that support for
reading container logs from the systemd journal is built.

Rename the Jmap member of the journald logdriver's struct to "vars" to
make it non-public, and to make it easier to tell that it's just there
to hold additional variable values that we want journald to record along
with log data that we're sending to it.

In the client, don't assume that we know which logdrivers the server
implements, and remove the check that looks at the server.  It's
redundant because the server already knows, and the check also makes
using older clients with newer servers (which may have new logdrivers in
them) unnecessarily hard.

When we try to "logs" and have to report that the container's logdriver
doesn't support reading, send the error message through the
might-be-a-multiplexer so that clients which are expecting multiplexed
data will be able to properly display the error, instead of tripping
over the data and printing a less helpful "Unrecognized input header"
error.

Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
2015-09-11 16:50:03 -04:00
David Calavera
f1412f2942 Remove doc that doesn't apply to Journald.
Signed-off-by: David Calavera <david.calavera@gmail.com>
2015-08-09 22:12:42 -05:00
Morgan Bauer
ccbe539e86
golint fixes for daemon/logger/*
- downcase and privatize exported variables that were unused
 - make accurate an error message
 - added package comments
 - remove unused var ReadLogsNotSupported
 - enable linter
 - some spelling corrections

Signed-off-by: Morgan Bauer <mbauer@us.ibm.com>
2015-07-29 13:09:39 -07:00
wlan0
9b782d3af3 add support for maximum log size, and max number of log files
Signed-off-by: wlan0 <sidharthamn@gmail.com>
2015-07-02 06:26:06 -07:00
John Howard
655a58e27b Windows: Factor out syslog and journald
Signed-off-by: John Howard <jhoward@microsoft.com>
2015-05-14 10:34:09 -07:00
Ahmet Alp Balkan
3a8728b431 daemon: Logging drivers refactoring
- noplog driver pkg for '--log-driver=none' (null object pattern)
- centralized factory for log drivers (instead of case/switch)
- logging drivers registers themselves to factory upon import
  (easy plug/unplug of drivers in daemon/logdrivers.go)
- daemon now doesn't start with an invalid log driver
- Name() method of loggers is actually now their cli names (made it useful)
- generalized Read() logic, made it unsupported except json-file (preserves
  existing behavior)

Spotted some duplication code around processing of legacy json-file
format, didn't touch that and refactored in both places.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-05-12 19:11:52 +00:00
Lars Kellogg-Stedman
869ecba652 journald log driver: use CONTAINER_ID field for container id
This patch modifies the journald log driver to store the container ID in
a field named CONTAINER_ID, rather than (ab)using the MESSAGE_ID field.
Additionally, this adds the CONTAINER_ID_FULL field containing the
complete container ID and CONTAINER_NAME, containing the container name.

When using the journald log driver, this permits you to see log messages
from a particular container like this:

    # journalctl CONTAINER_ID=a9238443e193

Example output from "journalctl -o verbose" includes the following:

    CONTAINER_ID=27aae7361e67
    CONTAINER_ID_FULL=27aae7361e67e2b4d3864280acd2b80e78daf8ec73786d8b68f3afeeaabbd4c4
    CONTAINER_NAME=web

Closes: #12864
Signed-off-by: Lars Kellogg-Stedman <lars@redhat.com>
2015-04-30 10:42:27 -04:00
Dan Walsh
364287b741 Add journald as a supported logger for containers
Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)
2015-04-21 12:57:54 -04:00