Hello. I have recently completely reinstalled one of our server machines
and everything is running great. While doing `netstat` using -p, I
noticed something odd; there are a couple of processes whose name does
not show. It is left blank (see below.) The processes that exhibit this
behavior are otherwise running just fine and doing their things, but why
don't their names get returned properly?


So far I've noticed this happens to named (bind), smbd (samba), mysqld
and children of httpd (apache) (while the main httpd process shows
fine.)

For instance (lines wrapped for readibility):

# netstat -tunp | egrep '/ '
tcp 0 0 192.168.0.4:80 192.168.0.2:2318
ESTABLISHED 13668/

# netstat -tunlp | grep httpd
tcp 0 0 0.0.0.0:80 0.0.0.0:*
LISTEN 13385/httpd


When normally I would see something like:

# netstat -tunap | grep ssh
tcp 0 0 0.0.0.0:22 0.0.0.0:*
LISTEN 1801/sshd
tcp 0 0 192.168.0.4:22 192.168.0.2:3020
ESTABLISHED 29676/sshd
tcp 0 0 192.168.0.4:22 192.168.0.8:1195
ESTABLISHED 22274/sshd


And if I lookup one of the nameless PIDs via `ps` it shows the command
without path and surrounded by square brackets, like the following:

# ps ax f | grep httpd | grep -v grep
13385 ? S 0:02 /usr/local/apache/bin/httpd -DHAVE_ (snip..)
13656 ? S 0:00 \_ [httpd]
13663 ? S 0:04 \_ [httpd]
13664 ? S 0:02 \_ [httpd]
13665 ? S 0:09 \_ [httpd]
13666 ? S 0:05 \_ [httpd]
13668 ? S 0:07 \_ [httpd]
13669 ? S 0:02 \_ [httpd]
13670 ? S 0:02 \_ [httpd]
13671 ? S 0:08 \_ [httpd]
13672 ? S 0:03 \_ [httpd]
13906 ? S 0:01 \_ [httpd]


Where I would normally expect something similar to:

# ps ax f | grep ssh | grep -v grep
1801 ? S 0:08 /usr/sbin/sshd
29676 ? S 0:11 \_ /usr/sbin/sshd
22216 ? S 0:05 \_ /usr/sbin/sshd
22274 ? S 0:01 \_ /usr/sbin/sshd



I am very interested in what could be causing this, and what a command
in side []'s actually means. I have been
administrating many different Linux systems over the past 6 or so years
and have never seen this sort of thing with netstat before.

Thanks greatly for any insight and info on this.

--
szr

On 02/14/2008 11:31 PM, szr wrote:
> And if I lookup one of the nameless PIDs via `ps` it shows the command
> without path and surrounded by square brackets, like the following:
>
> # ps ax f | grep httpd | grep -v grep
> 13385 ? S 0:02 /usr/local/apache/bin/httpd -DHAVE_ (snip..)
> 13656 ? S 0:00 \_ [httpd]
> 13663 ? S 0:04 \_ [httpd]
> 13664 ? S 0:02 \_ [httpd]
> 13665 ? S 0:09 \_ [httpd]
> 13666 ? S 0:05 \_ [httpd]
> 13668 ? S 0:07 \_ [httpd]
> 13669 ? S 0:02 \_ [httpd]
> 13670 ? S 0:02 \_ [httpd]
> 13671 ? S 0:08 \_ [httpd]
> 13672 ? S 0:03 \_ [httpd]
> 13906 ? S 0:01 \_ [httpd]
>
>
> Where I would normally expect something similar to:
>
> # ps ax f | grep ssh | grep -v grep
> 1801 ? S 0:08 /usr/sbin/sshd
> 29676 ? S 0:11 \_ /usr/sbin/sshd
> 22216 ? S 0:05 \_ /usr/sbin/sshd
> 22274 ? S 0:01 \_ /usr/sbin/sshd
>
>
>
> I am very interested in what could be causing this, and what a command
> in side []'s actually means. I have been
> administrating many different Linux systems over the past 6 or so years
> and have never seen this sort of thing with netstat before.
>
> Thanks greatly for any insight and info on this.

Hi.

If you were administrating many differen Linux system over the past 6 or
so years, you should be able to use man pages.

From the ps manpage (Section: STANDARD FORMAT SPECIFIERS):

command with all its arguments as a string. Modifications to the
arguments may be shown. The output in this column may contain spaces. A
process marked <defunct> is partly dead, waiting to be fully destroyed
by its parent. Sometimes the process args will be unavailable; when this
happens, ps will instead print the executable name in brackets. (alias
cmd, command). See also the comm format keyword, the -f option, and the
c option. [...]


check your processes with 'ps aux', and observe what the VZS, RSS and
STAT colums say.

Where:

VSZ virtual memory size of the process in KiB (1024-byte units).
Device mappings are currently excluded; this is subject to change.
(alias vsize).

RSS resident set size, the non-swapped physical memory that a task
has used (in kiloBytes).

STAT multi-character process state. See section PROCESS STATE CODES
for the different values meaning. See also s and state if you just want
the first character displayed.

If VSZ an RSS is 0 it technically takes up no space in Memory.
STAT flags are of particular interest:

PROCESS STATE CODES
Here are the different values that the s, stat and state output
specifiers (header "STAT" or "S") will display to describe the state of
a process.

D Uninterruptible sleep (usually IO)
R Running or runnable (on run queue)
S Interruptible sleep (waiting for an event to complete)
T Stopped, either by a job control signal or because it is being traced.
W paging (not valid since the 2.6.xx kernel)
X dead (should never be seen)
Z Defunct ("zombie") process, terminated but not reaped by its parent.

For BSD formats and when the stat keyword is used, additional characters
may be displayed:
< high-priority (not nice to other users)
N low-priority (nice to other users)
L has pages locked into memory (for real-time and custom IO)
s is a session leader
l is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
+ is in the foreground process group

All that information is taken from the ps manpage, you could've accessed
that very easily.

Regards,
--polemon

polemon wrote:
> On 02/14/2008 11:31 PM, szr wrote:
>> And if I lookup one of the nameless PIDs via `ps` it shows the
>> command without path and surrounded by square brackets, like the
>> following:
>>
>> # ps ax f | grep httpd | grep -v grep
>> 13385 ? S 0:02 /usr/local/apache/bin/httpd -DHAVE_
>> (snip..) 13656 ? S 0:00 \_ [httpd]
>> 13663 ? S 0:04 \_ [httpd]
>> 13664 ? S 0:02 \_ [httpd]
>> 13665 ? S 0:09 \_ [httpd]
>> 13666 ? S 0:05 \_ [httpd]
>> 13668 ? S 0:07 \_ [httpd]
>> 13669 ? S 0:02 \_ [httpd]
>> 13670 ? S 0:02 \_ [httpd]
>> 13671 ? S 0:08 \_ [httpd]
>> 13672 ? S 0:03 \_ [httpd]
>> 13906 ? S 0:01 \_ [httpd]
>>
>>
>> Where I would normally expect something similar to:
>>
>> # ps ax f | grep ssh | grep -v grep
>> 1801 ? S 0:08 /usr/sbin/sshd
>> 29676 ? S 0:11 \_ /usr/sbin/sshd
>> 22216 ? S 0:05 \_ /usr/sbin/sshd
>> 22274 ? S 0:01 \_ /usr/sbin/sshd
>>
>>
>>
>> I am very interested in what could be causing this, and what a
>> command in side []'s actually means. I have been
>> administrating many different Linux systems over the past 6 or so
>> years and have never seen this sort of thing with netstat before.
>>
>> Thanks greatly for any insight and info on this.
>
> Hi.
>
> If you were administrating many differen Linux system over the past 6
> or so years, you should be able to use man pages.

I am very well aquainted wit hthe man pages, thank you very much.


> From the ps manpage (Section: STANDARD FORMAT SPECIFIERS):
>
> command with all its arguments as a string. Modifications to the
> arguments may be shown. The output in this column may contain spaces.
> A process marked <defunct> is partly dead, waiting to be fully
> destroyed
> by its parent. Sometimes the process args will be unavailable; when
> this happens, ps will instead print the executable name in brackets.
> (alias cmd, command). See also the comm format keyword, the -f
> option, and the
> c option. [...]

I know all about this. Thank you anyways though. :-)


The problem, which I have solved, is do to a ptrace bug in the kernel I
had been running:

https://bugzilla.redhat.com/show_bug.cgi?id=88047

So I grabbed the latest kernel in the 2.4 branch (2.4.36), imported my
previous config, applied some patches so other things we use work
properly, and presto, no more ptrace bug. Everything works great.


Now as a side question, I'm thinking about migrating all our servers to
2.4 to 2.6 kernels. Obviously I'll be doing al lthe necessary research
before going and doing it. I would be interested in any advise on the
matter you or anyone else could give based on your experiences, as one
of the few things I haven't done is migrate a 2.4 system to a 2.6 and
from what I've read in posts and articles over the years on occasion is
that it's nto as clear cut as going from one 2.4.x top another 2.4.x

I don't mind experiementing either, as I keep all the past working
kernels as secondary boot options, as well as a cloned boot partition
and backup discs a plenty, and ditto for /lib/modules (not to mention
backups of the main file systems and such.)

-szr