Writing to internal FLASH on Arduino

History

In LCD88 I use internal flash as storage for model definitions. Of course, these should be editable without reflashing all firmware:-)

On AVR chips writing to flash is only possible from bootloader section, so I needed such feature in my bootloader.
I adapted my old X-MODEM bootloader to provide some entry points to functions writing flash. But because of using X-MODEM as communication protocol, using it was quite difficult for ordinary people. Then, I came up with idea that bootloader compatible with Arduino could solve this accessibility issue.
Nowadays, all things that can be invented, are invented and done by some people, so I started to find ready-made bootloader with all needed features. Most promising was Optiboot, but it lacked possibility to use it for writing flash by application.

Time was passing by and I was very surprised that I can’t find ready-made solution. More surprising was that on Arduino forum writing to flash by application was perceived as ‘mission impossible’. Every topic about it ended by failure or with getting external memory for storage (flash, fram or simply SD card), or advice to write your own bootloader:-)

My conclusion after all this research was: I must write my own bootloader or contribute to some other project to add such functionality. Then I gave a first look at the code of Optiboot, it’s issues and I found issue #52 with one promising patch at the end. But this patch didn’t fit to my needs, as I need fill-erase-write sequence to not using RAM as buffer, and it doesn’t do that that way.

A year passed by and I returned to Optiboot. This time I was going to do it by mysef. It was time of learning C, hacks used in avr-gcc, bugs in different verions of avr-gcc, learning Github flows, branching, merging, even writing in Arduino as I needed to provide some working example of using this feature. But finally I made it! ๐Ÿ™‚

Writing to flash

Optiboot with possibility to write to flash by applications could be downloaded from my ‘supermaster’ branch (also with some fixes not available yet in upstream): https://github.com/majekw/optiboot/tree/supermaster. It includes also example Arduino sketch (flash_program, heavily commented) writing to flash using new Optiboot.

So, feel free to use this long waited feature in Arduino world, and spread new, now better Optiboot bootloader.

Posted in Electronics | Tagged , , | 1 Comment

dm-cache in Slackware

bcache vs. dm-cache

I tested bcache about year ago on my spare/backup server and it worked quite fine. But since then every newer kernel had new issues, even on mailing list there were suggestions that bcache should be marked back as experimental. So, after some time I abandoned bcache and started to wait for something more stable and configuration friendly. In the same time dm-cache was also available, but it’s setup was quite challenging and dangerous (using dmsetup and raw calculated numbers – very easy way to make a horrible mistake). There was also quite friendly way to manage dm-cache using LVM2 tools, but only in plans ๐Ÿ™‚

Right now, LVM2 have all promised featured for managing dm-cache, so I gave it a try ๐Ÿ™‚

Just to sum up my experience:

  • bcache
    • + quite good documentation about cache policies, monitoring interface, disk format, disaster recovery etc.
    • + cache works on block device layer, so you can cache whole PV device regardless how many LV you have on top of it (one cache for all)
    • + no need for resize as you are caching whole physical device
    • + you can tweak cache parameters on the fly, including cache policy (writethrough/writeback)
    • – you can’t convert backing device on the fly to cached with data on it
    • – unstable
    • ~ doesn’t require LVM support
  • dm-cache
    • + stable
    • + easy to setup
    • + you can convert any LVM volume to use cache or not to use it, online, without any problem
    • – one cache for one LV, so if you have multiple LVs and you want to cache it all, you’d need to divide cache device (ssd disk) to many small chunks for every LV – space wasted
    • – very brief documentation – enough to set up things, but without any info about how to tune or monitor it
    • – resizing LVs must be done by uncaching, resizing and caching again, so you loose all cache contents
    • ~it’s part of LVM

Requirements

It’s time to get back to Slackware. Current version is 14.1, so everything described here is based on this version.

Recent kernel

Stock 3.10.17 kernel is little too old for good dm-cache work, so I recommend to get something at least from 3.14 series. You could compile it yourself or grab from slackware-current tree which have currently 3.14.33.

Recent LVM2

LVM2 version shipped with Slackware 14.1 is almost last version without dm-cache support, so there is a need to grab latest version and compile it. For complete support you need also thin-provisioning-tools with fsck-like tools for dm-cache.

Hard way (compile everything yourself)

  1. Get sources of LVM2 package from ftp://ftp.osuosl.org/pub/slackware/slackware64-14.1/source/a/lvm2/
  2. Get more recent LVM2 sources from ftp://sources.redhat.com/pub/lvm2/releases/LVM2.2.02.116.tgz
  3. Get thin-provisioning-tools from https://github.com/jthornber/thin-provisioning-tools/archive/v0.4.1.tar.gz and save it as thin-provisioning-tools-0.4.1.tar.gz
  4. Get my patch for SlackBuild from http://majek.mamy.to/dm-cache/slackware-lvm2-dm-cache.patch and apply it in lvm2 directory: patch -p0 < slackware-lvm2-dm-cache.patch
  5. Make package:sh lvm2.SlackBuild
  6. Upgrade lvm2 package with new one found in /tmp
  7. If you want cached root volume, you'd need also to patch mkinitrd and rebuild initrd.
    Patch: slackware-mkinitrd-dm-cache.patch.
    Apply:cd /sbin
    patch -p0 < /somewhere/slackware-mkinitrd-dm-cache.patch
  8. Use it :-)

Lazy way

Just get my lvm2 package and install it. You may also need mkinitrd patch (see above).

Configuration

Start is easy and well documented in man lvmcache.
Just quick howto assuming that you have volume group named vg_group, your physical cache device (ssd) is /dev/sdb1 and you want to cache volume named lv_home.

  1. First, cache/ssd disk must be part of volume group:vgextend vg_group /dev/sdb1
  2. Create cache pool on fast device (default mode is writethrough - safer but slower, so for writeback, there is a need to add --cachemode option):lvcreate --type cache-pool --cachemode writeback -L 10G -n cacheX vg_group /dev/sdb1
  3. Attach cache to logical volume:lvconvert --type cache --cachepool vg_group/cacheX vg_group/lv_home
  4. Enjoy ๐Ÿ™‚
Posted in Linux | Tagged | Leave a comment

Christmas tree lamps with Arduino and WS2811 leds :-)

Somewhere at the beginning of November I discovered RGB LEDs with embedded controller: WS2812 and others. Idea was so cool for me that I had to own one to play with it ๐Ÿ™‚
These chips became so common, that there are even put inside LEDs, like on this WS2812 diode:

WS2812 RGB LED

Considering time of the year, it was obvious to me to make whole string of lamps for Christmas tree. Then I found on Aliexpress ready made chains of such LEDs. ๐Ÿ™‚
So, my LEDs aren’t with chips inside, but with small PCBs sealed with some kind of hard silicone – maybe not so elegant as chip integrated inside LED, but cheaper – which counts if you buy 50 of them :-):

WS2811 RGB LEDs on Christmas tree

Day before Christmas I still haven’t done anything with these light, so it was last chance to do something useful with it.

Connecting everything was quite easy as there are only 3 components: 5V power supply, Arduino Pro Mini board and LED chain.

Programming protocol for WS2811 was more challenging – generated waveform must be very accurate, with some time intervals taking only 8 cpu Atmega328 clocks (at 16MHz). So, I chose assembler to do it (as usual ๐Ÿ™‚ ).
To be more hardcore, it was 3am when I started debugging ๐Ÿ™‚ Honestly, without my new oscilloscope, finding these 3 mistakes I’ve made, could’ve took hours. Especially that Chinese guy who sells these LEDs described them as ‘we sent WS2811. Or we may sent UCS1903’ – two different possible chips (!) with two different protocols and no one knows which version I’ve got.
Luckily for me. it was WS2811, so first guess and code worked.

I added few simple routines to move light across whole chain of LEDs and put everything on Christmas tree.

All code for this ‘one night project’ you could find on my Github page: https://github.com/majekw/wsled

This is how it looks like on my Christmas tree:

Posted in Electronics | Tagged , , , | 2 Comments

Initrd in Slackware

There is small simple howto with Slackware package mkinitrd, man page says a little bit more, more is kind of mystery ๐Ÿ™‚
I switched from full kernel (in fact, my own compiled kernel) to initrd few years ago and since then I had learned a lot about it (sometimes in hard way) ๐Ÿ™‚

So, this is my way (a little different than in man and readme) of doing initrd.

Installation

During Slackware installation you need to install:

  • kernel-generic (not kernel-huge!)
  • kernel-modules
  • mkinitd

and of course other packages you would like to have ๐Ÿ™‚
Remember NOT to install kernel-huge package.

Install everything, setup lilo, timezones etc as usual. At the end DON’T reboot at the end of setup – system isn’t finished yet!

Now, it’s time to prepare new initrd.

  1. Chroot to /mnt (where new system should be already mounted)
  2. Make copy of default mkinitrd config: cp /etc/mkinitrd.conf.sample /etc/mkinitrd.conf
  3. Edit new /etc/mkinitrd.conf with your favourite editor and set few options:
    • uncomment and set MODULE_LIST and ROOTFS to type of your root filesystem, for example ROOTFS="ext3"
      MODULE_LIST="ext3"
      You don’t need to worry about all related modules, they will be added automatically.
    • if you use software raid, uncomment and set RAID="1"
    • if you use lvm or encryption, uncomment and set LVM="1"
    • I strongly recommend to set SOURCE_TREE and OUTPUT_IMAGE to some more unique names, for example SOURCE_TREE="/boot/initrd-tree-3.10.12"
      OUTPUT_IMAGE="/boot/initrd-3.10.12.gz"
      You don’t have to do this if you don’t want to play with custom kernels etc.
  4. make new initrd with mkinitrd -F
  5. add initrd image to lilo.conf adding it just after ‘image’ line, so it will look like this:image = /boot/vmlinuz-generic-3.10.12
    initrd = /boot/initrd-3.10.12.gz
  6. get out of chroot and rebuild lilo with:mount --bind /dev /mnt/dev
    mount --bind /proc /mnt/proc
    lilo -r /mnt

Right now you could reboot system and enjoy new Slackware with initrd:-)

What’s an advantage of this way instead of running mkinitrd with many bizarre options? See next part.

Normal usage

Sometimes you need to change kernel, upgrade or something like that.

There are 2 ways of maintaining Slackware with initrd:

  1. Use one initrd for all kernels
  2. Use separate initrd for every kernel or even every entry in lilo

One initrd for all

At start, make sure that you have commented out CLEAR_TREE or set it to 0 in mkinitrd.conf!

After installation of new kernel and running still from old one, you need to to set KERNEL_VERSION to version of your new kernel, for example KERNEL_VERSION="3.17.4"
Then just run mkinitrd -F
That’s all about initrd – now image has required modules for both/all your kernels.
Of course you need new entry in lilo.conf for new kernel with exactly the same initrd as old one.
If you are making kernel upgrade (replace old kernel package) you don’t even need to touch lilo.conf. But in both cases, you must run lilo at the end.

From time to time, you could clean initrd tree from old modules from kernel you don’t use anymore. In this case just remove all directory with that modules from /boot/initrd-tree/lib/modules/old-kernel-version and regenerate initrd using as usual mkinitrd -F. And rerun lilo of course ๐Ÿ™‚

Now, you probably see advantage of using mkinitrd.conf instead of remembering all those weird switches you used last time to generate this image.

Separate initrd for every kernel

This way have a little sense if you only use one root filesystem and only stock kernel which you sometimes only upgrade.

Separate initrd is even easier than ‘one for all’ attitude.
One drawback is that mkinitrd with -F flag only reads one default /etc/mkinitrd.conf file, so each time you have to copy saved configuration back to this file.
So, for first kernel you should customize at least SOURCE_TREE, OUTPUT_IMAGE and KERNEL_VERSION. Generate new initrd with mkinitrd -F, then make backup of mkinitrd.conf to for example mkinitrd.conf.3.10.17.
For next kernel with different options customize mkinitrd.conf again, create new initrd, make backup and so on.
That way you could have different initrds and saved all options to regenerate each of them. It’s safe attitude as playing with new kernel and initrd configuration you always have working old one.

Ok, that was easy part:-)
Next time I’ll cover how to combine initrd with encrypted root and even root on encrypted lvm ๐Ÿ™‚

Update 2015.02.20

I found quite good article on the same topic at Slackware Documentation Project. So, it looks like i didn’t do so much research before writing this post ๐Ÿ™‚

Posted in Linux | Tagged , | Leave a comment

Picture thief

Recently I noticed in server’s logs that one of my blog pictures become very popular:
SDR# in action
It’s from this post: RTL2832U spectrum analyzer โ€“ part 2

Sometimes people embed my schematic or pictures on forums to enhance discussion. But this time one seller from Italy used my picture in his eBay offer. Moreover, he embedded my (and not only my) full resolution picture there, so he is also using my server bandwidth for free. Everything without a word.

There are many ways of dealing with that type of abuse, but I decided to use it as free promotion ๐Ÿ™‚

I prepared new edited version of this picture (also smaller one):
sdrsharp-ebay

And set new rule in .htaccess to redirect all requests from eBay page to new file:
RewriteCond %{HTTP_REFERER} ebaydesc\.com/|ebay\.it/
RewriteRule ^wp-content/uploads/2013/06/sdrsharp\.png /wp-content/uploads/2014/12/sdrsharp-ebay.jpg [R=302,L]

Now, his offer shows my new picture ๐Ÿ™‚

Posted in Linux | Tagged | Leave a comment