Bob Bob wrote:
> I am trying to squeeze a bit more I/O bandwidth out of a video
> processing box w/out changing the hardware. [...]
> [...]
> Was considering playing with buffer flushing times or maybe the file
> system itself. It is running vanilla reiser.

I'm not an expert, but over the years the usual thing I read about
Reiser vs ext3 is that Reiser is very fast with *small* files while ext3
is very fast and uses less CPU with *big* files.

Bob Bob wrote:

> Hi Nikos
>
> Okay thanks for that, something I didnt know!
>
> I am mostly manipulating 80-100KByte files so I guess that qualifies as
> small.
>
> I note there is disk elevator adjustment that I may look at. (ie wait
> for a list of read and writes to gather before making the heads move and
> do it)
>
> OBTW already using a RAM disk for preprocessing.

With 256Mb RAM, I'm surprised that it will stagger along even without the
RAM disk! Note that you have 2x RAM in swap - your only solution is going
to HAVE to include more RAM - you're putting 15 pounds of crap into a 5
pound bag so you can bet that you are spending more time swapping in/out
than actually processing anything.

Depending on your mb, you look to need at least 1GB RAM to gain any real
performance. Before you run out an buy Ram, check the mb specs and see
what it will support. Many of the old PIII boards don't support 512/1024MB
Ram sticks - I have a couple of Dell GX boxes that won't deal with more
than 256Mb sticks for a total possible 512Mb max. Also watch the
spped/voltage spec of the memory if you decide to acquire more - most of
the bargain stuff now is 3.3VDC PC4300+ spec and won't like your slower bus
and refresh.

There really is no substitute for enough memory for the job.

--
Will Honea
** Posted from http://www.teranews.com **

Will Honea wrote:
> Bob Bob wrote:
>
>> Hi Nikos
>>
>> Okay thanks for that, something I didnt know!
>>
>> I am mostly manipulating 80-100KByte files so I guess that qualifies as
>> small.
>>
>> I note there is disk elevator adjustment that I may look at. (ie wait
>> for a list of read and writes to gather before making the heads move and
>> do it)
>>
>> OBTW already using a RAM disk for preprocessing.
>
> With 256Mb RAM, I'm surprised that it will stagger along even without the
> RAM disk!

I didn't pay enough attention and missed that there's only 256MB RAM!
You are right; with anything less than 1GB you can't expect to get
performance on a video-editing box. Also, I'm a bit surprised to hear
"mostly manipulating 80-100KByte files". What kind of video processing
are we talking about here exactly? Videos usually mean multi-GB file sizes.

Bob Bob wrote:
> [...]
> I dont think RAM will help all that much. The actual I/O rate numbers
> look like;

RAM always helps. More RAM = more buffer. Without RAM the data has to
be written to disk sooner, and that's slow. Btw, the fact that swap is
used at all on a machine that is not multi-user means it's time to get
more RAM. The machine should never use swap at all. If you really
can't add more RAM, you should move the swap partition to a hard drive
that isn't used much. Using swap results in disk thrashing and disk I/O
suffers (greatly).


> as the capture rate. hdparm -tT for one of the RAID0 devices shows about
> 20Mbytes/sec buffered disk reads. Maybe there is a lot more I/O going on
> than I thought?

If you really have millions of files, it is crucial to not have more
than 700.000 per directory. Google for "reiserfs millions of files".

But again, more RAM (another 256MB wouldn't hurt) and a faster disk
(access time is more important here than throughput) will help. And RAM
is cheap these days, so why not just try.

Bob Bob wrote:

> Will/Nikos
>
> Tnxs for your feedback. I tried to keep the question simple, but see I
> need to elaborate!
>
> I have certainly allocated 2x RAM to swap but note that it is barely
> used. To repeat the numbers;
>
> Memory: Total Used Free Shared Buffers
> Mem: 255416 138680 116736 0 90028
> Swap: 522072 8384 513688
>
> Only a very small amount of swap is used and there is still 90MB of
> buffer space. Depending on the weather that buffer space goes up to
> about 120MB. This will of course all be disk cache.
>
> I am sorry I didnt explain the system well enough. It isnt video editing
> and it isnt interactive. It is more a surviellance system. I capture
> from 6 IP cameras, jpg images at around 5-6FPS and 640x480. Each jpg
> compressed filesize is in the order of 50-100K. The current capture uses
> wget and works surprisingly well. Since I use noclobber I then have to
> rename the files to a timestamped name. This obviously adds time to
> processing that has to happen in real time, so I shell/fork out & to do
> this. The right way is of course to write code that captures and writes
> direct to a stamped filename. (I am aware that some GPL projects area
> available already do this) That will be the next step if I can't resolve
> the I/O bottleneck. The renaming process is set to inhibit the wget
> process if it runs overtime, which it does as the I/O load gets higher.
> The result is that their are gaps in the capture stream. (I use an input
> and output directory and swap them when renaming is finished)
>
> After capturing I use mjpegtools (mpeg2enc etc) to create 1FPS MPEG2
> videos from the individual images. (It also does motion detect) These
> are used to rough catch thieving events after which the 5FPS images are
> checked and/or a 5FPS MPEG is created for the date/time in question. We
> have some classic shots of people tucking shop items into their shorts,
> pants, pockets and even socks! (Its a non profit org "thrift" store)
> Whether talking a 1FPS or 5FPS MPEG video they are between 20 and 50MB
> in size. (They are of varying time lengths)
>
> So it isnt as badly loaded as you might think. At any one time there are
> a lot of images on the server, maybe 1-2 million for 2-3 days. They are
> broken up by camera to make find/sorts a little faster and I think NFS
> also suffers with the large number of files per directory. I cant do an
> ls for example as shell expansion runs out of space. (I use find and
> xargs) During the development (one might say playing) process I hit the
> mpeg creation limit problem first. I needed 30 hours to process the days
> data. I then clustered two more machines in and read up/implemented
> reducing the cpu grunt required for mpeg2enc. This was all originally at
> 1FPS. I now find I have hit an I/O rather than cpu limit trying to
> increase the frame rate.
>
> Apologies for the length.
>
> I dont think RAM will help all that much. The actual I/O rate numbers
> look like;
>
> The capture side is maybe 3Mbytes/sec but that goes to the ramdisk. The
> moving/renaming will be around the same rate but of course writes to the
> real RAID0 disk. The 1FPS mpeg creation for an hours worth of data per
> cameras takes say 15 minutes. Thats about 350MB of reads or
> 0.5Mbytes/sec. There are three of these running plus a bit of other I/O
> that wouild on average maybe an extra 50% (I have to recreate reference
> images and that uses "convert") Say all up 3MBytes/sec. About the same
> as the capture rate. hdparm -tT for one of the RAID0 devices shows about
> 20Mbytes/sec buffered disk reads. Maybe there is a lot more I/O going on
> than I thought? Have to look at the mpeg creation script some more..
>
> Would still like to hear your ideas.
>
> Cheers Bob
>
> Nikos Chantziaras wrote:
>>> With 256Mb RAM, I'm surprised that it will stagger along even without
>>> the RAM disk!
>> What kind of video processing
>> are we talking about here exactly? Videos usually mean multi-GB file
>> sizes.

I'm with Nick - I read what I expected instead of what you wrote. Now I'm a
bit suspicious as well, even with you expanded explanation but before we
get too far gone I'd sure like a better look at what you are doing here - I
have a similar project in the works where your system sounds like a very
good fit since there is a very cheap (about $160 for 3 wireless cameras and
a receiver that scans the 3 cameras on either a fixed schedule or on
command). I think we had a brief exchange on this a few weeks back.

I started developing real time software back when we used analog computers
for anything faster than about 1 hz. Given the cost of RAM circa the late
60's - early 70's we got pretty good at locating choke points and, believe
it or not, RAM disk can be a killer! This is especially true as the number
of files increases. What happens is that the disk cache system gets choked
for lack of room. The latency of disk data (and your RAM disk is cached as
well unless you've done some fiddling) is such that the cache is constantly
being invalidated because it's too small! You can see this with a highly
threaded compiler and build system (OS/2 was great for this) on a huge
project with many, many source files producing a huge flow of object files
which in turn get piped to a linker creating a large number of linked files
(dll, executable, etc.). I haven't really stressed gcc, but I would expect
a similar response. I've run the tests many times over the years and
allowing the disk cache to use as much memory as it wants has won hands
down every time. That even goes back to 4 mhz PCs with MFM/RLL drives.
Once your data stream exceeds the disk cache capacity you actually increase
the disk access count by nearly two to one as the over loaded cache is
flushed and re-loaded for pretty much every r/m/w operation. I don't know
what tools may be available for Linux but one that shows cache hit rates
would probably make the problem glaringly obvious - especially for that
processing loop. Try disabling the RAM disk and see if it helps.

A second trick I've used over the years that may help is to offload the disk
storage to a network drive. I've had cases where that made a big
difference even with 10mbs ethernet on slower machines. What you trade off
here is that the disk cache is offloaded to another box so the excessive
thrashing is reduced.

--
Will Honea
** Posted from http://www.teranews.com **

Will Honea wrote:
> There really is no substitute for enough memory for the job.

I know a girl who doesn't have any memory, but she has a nice rack.
Great subsitute to be able to keep her job.

houghi
--

You are standing at the end of a road before a small brick building.
Around you is a forest. A small stream flows out of the building and
down a gully.

Nikos Chantziaras wrote:
> If you really have millions of files, it is crucial to not have more
> than 700.000 per directory. Google for "reiserfs millions of files".

If the issue is ReiserFS, I would sugest using a different file system.
No, I do not know which one.

houghi
--

You are standing at the end of a road before a small brick building.
Around you is a forest. A small stream flows out of the building and
down a gully.

houghi wrote:

> Will Honea wrote:
>> There really is no substitute for enough memory for the job.
>
> I know a girl who doesn't have any memory, but she has a nice rack.
> Great subsitute to be able to keep her job.

Now you're comparing acorns and melons ;) One has to define the job
descriptions to determine which is the substitute, although I am something
of a melon fan.

--
Will Honea
** Posted from http://www.teranews.com **

Bob Bob wrote:

> Hi Will
>
> Yes it may have been us talking about camera stuff!
>
> At the sake of possibly repeating myself, we bought two camera types
> from Trendnet, both wireless but one a steerable ball model. They are
> however vastly different in performance and quality. The steerable model
> has an atrociously bad definition image compared to the fixed one.
> They never appear sharp at all. (They are both 640x480) The steerable
> one however does MJPEG and if you use its FTP client you can send about
> 4-5FPS to a server. The fixed one however will only do about 0.7FPS in
> ftp client mode and streaming is done by sending multiple http auth/gets
> with different source ports rather than just reusing the one. The camera
> then sends back complete jpg's not mjpeg. Wget presents the same
> problem. ie if I just try to grab one URL at a time and loop it I cant
> get any faster than the 0.7FPS. If however I give it a list of 100URL's
> from a file it will do maybe 8-10FPS.
>
> This for me has actually been quiet interesting as it is a non profit/no
> pay project. If I was doing it for money I'd be real upset!
>
> Yep I remember XT's and ST225's! I however cut my teeth on a PDP11-44
> <grin>
>
> I originally went to a ramdisk to reduce latency but I can see your
> point now that using RAM for this instead of cache isnt so good. It is
> no biggy changing the script as I did my initial work w/out a ramdisk. I
> actually now wonder whether "free" includes a ramdisk size in "buffers".
>
> You're right too, I have never seen any linux tools for checking disk
> cache hits. In a case like mine it would be handy to say "always keep
> these files in cache" like those you mentioned (libraries etc). When
> working with moving files around once it is hardly worth caching them
> for a later read too. It is however handy to queue them up for one
> write. (My thoughts on using the elevator code)
>
> As it turns out I just acquired another P3/900 to rebuild for resale at
> the thrift store.. It has 512MB of RAM so I can swap that with the
> processor machines. I also came up with a method to avoid the
> file/timestamp renaming process. Kind of bad to make three changes at
> once, but if it works I'll be happy.
>
> Shall do some more work and get back to you with the results. Thanks for
> your input.
>
> OBTW do you know of any NFS limitations do do with large directories of
> files? I am nowhere near the 750,000/directory Reiser limit. Probably
> around 450,000 maximum.
>
> Bob
>
> PS My NG email address is valid if you want specific details on the
> system.

Thought the project rang a bell - I'll tuck the addy away and check with you
every now and then.

You're in the one area of Linux that I've never been satisfied with. You
really need a good multi-threading model with re-entrant coding all the way
through and the only PC OS I've ever found that could handle
multi-threading on that scale was OS/2. Like everything else, each OS has
it's strengths and weaknesses I guess.

You would be one of the folks that could benefit from a good RAID array <g>.

--
Will Honea
** Posted from http://www.teranews.com **

Bob Bob wrote:

> Hi Will
>
> It is early days yet but it is interesting to note that the IOwait has
> gone up over the last few days;
>
> Memory: Total Used Free Shared Buffers
> Mem: 515300 508808 6492 0 78856
> Swap: 522072 2708 519364
>
> Bootup: Sat Jul 19 18:26:12 2008 Load average: 8.61 8.28 8.08 2/87 2510
>
> user : 4:31:46.86 9.9% page in : 10554322
> nice : 0:00:00.00 0.0% page out: 188974664
> system: 3:13:05.10 7.0% page act: 6876054
> IOwait: 1d 0:07:38.61 52.7% page dea: 4897097
> hw irq: 0:31:44.90 1.2% page flt: 336263666
> sw irq: 0:18:47.73 0.7% swap in : 0
> idle : 13:02:43.16 28.5% swap out: 677
> uptime: 1d 21:45:50.86 context : 15110435
>
> It was around 37% last week. With such a short elapsed time though the
> 52% is probably not indicative of the true figure.
>
> I remember during onight processing Saturday that the Buffers went to
> 135000 or so.
>
> I decide to leave things as they were (ie not implement any other
> changes) as a kind of test. The changes I am thinking of doing reduce
> the capture latency in favour of after hours processing. As it turns out
> I *think* the inhibit/latency issue is slightly better. It is however
> still there!
>
> I note some comments too that procinfo might be kind of flawed. I havent
> investigated this in depth but heard it was something that went awray
> when kernel 2.6 came out...

I ran into a problem with an older box (Dell GX, 512Mb Ram, 700 mhz
processor) yesterday when doing some maintainence on a DB2 database I keep
there. It serves some relatively large tables - in the million row range -
and is adequate for normal operations but when I ran a stress test things
descended to the molasses range in a hurry. I had recently updated that
box from OS 10.2 to 11.0 so I checked the kernel settings. The kernel io
scheduler was "unconfigured" (using default settings) so I started playing.
The CFS scheduler was a disaster for this particular case but the AP
(anticipatory) scheduler gave me a significant bump in throughput by
comparison. Note that this was a case of an identifiable app with high,
continuous io demands and it slowed some other apps to a crawl but I can
live with that - the box exists pretty much for that database server
function so I'll take the hit. You might want to check the kernel settings
you have. I have to admit that this is an extreme test case but it does
show the importance of system tuning to the app at hand.

Another nit to tuck in your bag <g>. I would suggest that you look into
updating to 11.0 as a possible improvement - it feels faster all around.

--
Will Honea
** Posted from http://www.teranews.com **