Category Archives: howto

Log management with Graylog & Fluentd

A perfect match -Graylog & Fluentd

In the following post, I’ll describe how to quickly setup a docker multi container environment running Graylog and Fluentd. The result is a comprehensive log management platform that is able to collect log data from distributed applications.

Graylog?

Think of Graylog as an open source alternative to Splunk Enterprise, a log management platform for collecting, indexing, and analyzing both structured and unstructured data . Furthermore, you can configure email alerts for certain events and dashboards to monitor your applications, quickly.

Fluentd?

Fluentd is an open source data collector, an unified logging layer. It decouples data sources from backend systems by providing a unified logging layer in between. If your applications running within Docker containers, you might be interested in the OOTB logging driver for Fluentd.

Why do I need this?

There are a lot of use cases for such a comprehensive log solution. Especially, if log files are getting bigger and are distributed across multiple servers / applications, it can be quite time consuming analysing the logs.

In my case, I’m using a Scala logback Fluentd appender that forwards all log messages of the application to a Fluentd collector running on another server. Each application got it’s own tag like “applicationXY.prod” or “applicationXY.staging“, so we can differenciate the messages later on in Graylog.

The Fluentd collector or “Fluentd to Graylog forwarder” receives and forwards all log messages to Graylog where they got indexed and persisted.

You might wonder why I don’t send the log messages directly to Graylog? Fluentd has many advantages in terms of log  message handling. For example,  Fluentd supports log file or  memory buffering and failovers to handle situations where Graylog or another Fluentd node would go offline. This is what fluentd is really good at. For more information look at the fluentd out_forward or buffer plugin to get an idea of the capabilities.

Ok, ok.. How do I set this up?!

I put a multi docker container environment together, to speed up this process.

First, it builds a custom Fluentd to Graylog forward container, based on the official Fluentd container and a Graylog (gelf) plugin. Then it links it to the official docker all-in-one container for Graylog, which consists of Elasticsearch, mongodb, nginx and Graylog of course :-).

I’d recommend going the easy route and install docker-compose.

Then simply execute following command. This will download the all-in-one container, build the Fluentd-gelf forward container and link it all together.

Now, you should be able to access graylog on [container-IP]:9000 with credentials admin:admin. Go to [container-IP]:9000/system/inputs and launch a new Gelf UDP input with the default settings:

GELF - UDP

Graylog is now listening on port 12201 to receive messages from the Fluentd to Graylog container we built. The container expects messages in the Fluentd format (json) on its TCP input on port 24224 with a tag “gelf.app.XYZ and forwards them to Graylog.

You can now start logging with your Fluentd appender and setting up different streams and dashboards within Graylog by separating the log messages by its “gelf.app.XYZ” tags since they are also forwarded to Graylog.

For more detailed instructions, please check out the Readme.

Fluentd log appenders

Since this is application dependent, here are some links that might help.

 

 

 

 

 

 

How to build an Ambilight for every HDMI input source

In this post I’m going to show how you can configure your Hyperion Ambilight for every HDMI source. If you don’t have an Ambilight setup yet, I kindly refer you to my previous guide, which will give you an initial ambilight effect for the media center running on the Rpi.

This guide will then go one step further and enable the ambilight effect for all kind of HDMI input sources like PS3, XBOX, Chromecast etc.

We need to get the color information from an HDMI input signal. For this purpose, it’s necessary to transform the digital HDMI signal to an analog composite one with a converter. After this, we can grab the composite signal with an USB video grabber connected to the Raspberry Pi. Now we’re able to feed Hyperion with the color information by the video grabber.

Parts list

AVR Receiver It’s essential. Most TVs don’t offer an HDMI output
HDMI to Analog converter Speaka Professional HDMI / Composite Converter. There are several others on the market, make sure that you got one with 1080p support. Check this guide for more details about the available chipsets
2 port HDMI splitter My AVR features two HDMI outputs. Otherwise make sure you’ll get one!
USB Video Grabber (Easycap) Make sure you got a grabber with the STK1160 chipset, because it’s fully supported on Linux! Check here for detailed instructions, this can be tricky…I’d recommend a grabber with the UTV007 (Fushicai) chipset. It’s supported with OpenElec >=4.2.1 and features the best results for this purpose.
Composite cabel Just a standard video cinch cabel
Active USB Hub The Raspberry won’t be able to power most of the devices directly from it’s USB port.

Let’s wire it up. Connect the HDMI output from your AVR or from your HDMI splitter to the HDMI / Composite converter input. Use the composite cabel for connecting the USB video grabber with the Composite converter analog output. Plug the USB video grabber into the active USB hub, connect the hub to the Raspberry Pi.

If you already have a running media center then you might already have an analog video output on this machine. In this case, connect the analog output to the grabber, directly. Another possibility would be to go for an DisplayPort to composite converter that connects to the USB grabber. For both solutions, make sure that your existing media center is capable of playing your media to both outputs, simultaneously otherwise you’ll need to go for a 2 port HDMI splitter (without AVR) and HDMI to composite converter.

Here’s my setup connected, ready for video capturing from any HDMI input.

 

1. USB Video Grabber setup

Connect the USB Video Grabber to the Raspberry and type

Unfortunately, my device with “ID 1b71:3002” features a Fushicai UTV007 chip, which is only initial supported with kernel 3.11. This forced me to compile the kernel module myself. If you got the same problem, here’s my precompiled module “usbtv.ko” for Raspbmc with kernel 3.10.36. Make sure you first load the modules “videobuf2-vmalloc” and “videobuf2-core”, than load the module with insmod usbtv.ko.

OpenElec (>=4.2.1) has the driver for Fushicai UTV007 already included, therefore it will work out of the box. Remember, to enable ssh you have to boot OpenElec with TV connected. After that, you won’t need the installed XBMC/KODI anymore. It seems like the current Raspbmc release has the driver not included. SSH to openelec and install hyperion:

Then copy your hyperion.config.json to the shared SMB “Configfiles” folder from your host system. The shared “Configfiles” folder is actually mapped to /storage/.config on the Rpi filesystem. After a Rpi reboot, Hyperion should autostart and load the config from /storage/.config/hyperion.config.json automatically, though.

Remember that you have to set the library path for OpenElec setups before you can execute any hyperion command:

Stop / Restart hyperion manually:

Take a screenshot from the grabber with following command. Remember to kill hyperiond before you can execute this command properly!

For further instructions, make sure to checkout the documentation.

2. Hyperion configuration

Hyperion already features capturing color information from a USB video grabber. Add following configuration to your hyperion.config.json file to enable it.

To adjust the cropping for your setup, you have to change the values “cropLeft, cropRight, cropTop, cropBottom” and make sure “device” matches with your ls /dev/video* output!

Check the created screenshot.png, adjust the crop-* values till the grabbed input has no black borders left. Then put the calibrated values into your /etc/hyperion.config.json file.

Screenshot from my USB grabber input, I was able to get rid of the upper and lower borders by adjusting the cropTop and cropBottom values. I was able to get rid of the green lines after increasing “frameDecimation” to 8 but then a lot of frames were skipped and the LED’s were out of sync to the moving picture. So I set it back to 2.

If you only want to grab the signal from the v4l2 video grabber, make sure that you remove the “framegrabber” and xbmcVideoChecker sections.

Let’s test some videos! If you don’t like the results, try to fine tune your setup with different SignalThresholds or try to adjust the hsv settings. Make sure to read this comprehensive guide about color transformation.

When you switch the HDMI source on your AVR to non HDMI sources, like for example audio only devices, the LED’s will light up in blue. Blue is the default color if no video source was detected for many AVRs. You can disable the LED’s in this situation by setting the blue-threshold value to 1.0.

3. Conclusion

Now you should be able to use your Ambilight for any HDMI input you have connected to your AVR. If you have any question, feel free to ask in the comments section. Enjoy your unleashed Ambilight!

Use Apple’s USB SuperDrive with Linux

I’m really surprised and disappointed that Apple prevents us from using their USB SuperDrive with non Apple devices.

How to outsmart Apple’s firmware

Fortunately, with a little hack, we can awake the drive from its deep slumber. It’s required to send a “magic” byte sequence after the drive was connected. Thanks to “A Random Hacker” for figuring this out.

You have several options for making this work. In this post I’d like to unveil two of them.

Unlock with SCSI Generic (sg) driver

For communicating with the SCSI device directly we need the Linux SCSI Generic (sg) driver packages.

Lookup the device, it should be sr0 or sr1 by default depending on how many USB disc drives are currently attached. Check the output of following command to get a list off all device paths:

After you’ve the SuperDrive identified, we’ll send the magic sequence to the device.

Try to insert a disc, the drive should be awake now and start initialising the disc. For now the last step is necessary each time the drive is unplugged, so let’s automate it!

Custom udev rule

We’ll make us of the udev device manager. It runs as a deamon and receives events each time a device is initialised or removed. Furthermore, it features an extensible rule set for easy customising. Please check out this very good guide for further instructions.

Let’s write such a custom rule.

Add following rule definition.

This will do the “magic” each time a SuperDrive device is connected. To test the rule, disconnect the drive and connect it again, the drive should be unlocked, already.

Unlock with Superdrive-Enabler

Superdrive-Enabler is a little app that sends the magic byte sequence to a device.

Superdrive-Enabler for Raspberry Pi

I precompiled a binary for the Raspberry Pi in cases where you can’t or don’t want to install “sg3-utils”. Make sure that SuperDrive is connected via an active USB hub to the Pi. Copy the executable binary to your Raspberry with SMB or WGET.

Other distributions

Easily compile the superdrive-enabler source.

Custom udev rule

Let’s write a custom rule in a new *.rules file or separated by a line break in an existing rules file.

Add following rule definition.

This will trigger the “superdriveEnabler” app with the device path as parameter (e.g /dev/sr0) each time a SuperDrive device is connected. Reconnect the drive again and enjoy your CD/DVD collection with XBMC or any other media player!