I've this (let's say strange) problem in a communication between one Linux
server with kernel 2.6.18-6 (Debian) and one Windows Server 2008
Enterprise; the policy requires AH with x.509 certificates. If the Linux
machine tries beginning the communication (on e new connection) the Quick
Mode SA negotiation fails (but the phase 1, main mode negotiation
succedes). When the communication begins from the Windows server, all go
just fine and from that moment, obviously, the communication works in both
ways.
In the syslog I can read:
racoon: ERROR: mismatched ID was returned.
racoon: ERROR: failed to pre-process packet.
racoon: ERROR: phase2 negotiation failed.
In the Windows event viewer i have one failure for any one Main mode
nagotiation (failed or succeded):
Event ID: 4976
Task Category: IPsec Main Mode
Level: Information
Keywords: Audit Failure
Description:
During Main Mode negotiation, IPsec received an invalid negotiation packet.
If this problem persists, it could indicate a network issue or an attempt
to modify or replay this negotiation.
Any idea?
Thanks.

--
Lorenzo Vaina,
MCSA Windows Server 2003,
MCTS SQL Server 2005.
messaggi privati: http://www.vaina.it/posta.html

Am Tue, 08 Jul 2008 20:38:02 +0200 schrieb Lorenzo Vaina:

> I've this (let's say strange) problem in a communication between one Linux
> server with kernel 2.6.18-6 (Debian) and one Windows Server 2008
> Enterprise; the policy requires AH with x.509 certificates. If the Linux
> machine tries beginning the communication (on e new connection) the Quick
> Mode SA negotiation fails (but the phase 1, main mode negotiation
> succedes). When the communication begins from the Windows server, all go
> just fine and from that moment, obviously, the communication works in both
> ways.
> In the syslog I can read:
> racoon: ERROR: mismatched ID was returned.
> racoon: ERROR: failed to pre-process packet.
> racoon: ERROR: phase2 negotiation failed.
> In the Windows event viewer i have one failure for any one Main mode
> nagotiation (failed or succeded):
> Event ID: 4976
> Task Category: IPsec Main Mode
> Level: Information
> Keywords: Audit Failure
> Description:
> During Main Mode negotiation, IPsec received an invalid negotiation packet.
> If this problem persists, it could indicate a network issue or an attempt
> to modify or replay this negotiation.
> Any idea?
> Thanks.
>

check your phase2 proposals on debian.

Burkhard Ott scripsit:

> check your phase2 proposals on debian.

Thank you for your reply. I re-checked the sainfo stanza and it seems fine.
Oh, it's so simple:

sainfo anonymous
{
pfs_group modp2048;
lifetime time 15 min;
encryption_algorithm 3des,null_enc;
authentication_algorithm hmac_sha1;
compression_algorithm deflate;
}

I added null_enc as a fallback trying but obviously all I got was to load an
unsuitable module in my poor kernel memory.
I think that if something was wrong here, the opposite way communication
failed too.

Please, do you know with ID is referred to, in the stanza:
racoon: ERROR: mismatched ID was returned.
?

Thanks.

--
Lorenzo Vaina,
MCSA Windows Server 2003,
MCTS SQL Server 2005.
messaggi privati: http://www.vaina.it/posta.html

Am Wed, 09 Jul 2008 13:52:01 +0200 schrieb Lorenzo Vaina:

> Burkhard Ott scripsit:
>
>> check your phase2 proposals on debian.
>
> Thank you for your reply. I re-checked the sainfo stanza and it seems fine.
> Oh, it's so simple:
>
> sainfo anonymous
> {
> pfs_group modp2048;
> lifetime time 15 min;
> encryption_algorithm 3des,null_enc;
> authentication_algorithm hmac_sha1;
> compression_algorithm deflate;
> }
>
> I added null_enc as a fallback trying but obviously all I got was to load an
> unsuitable module in my poor kernel memory.
> I think that if something was wrong here, the opposite way communication
> failed too.
>
> Please, do you know with ID is referred to, in the stanza:
> racoon: ERROR: mismatched ID was returned.
> ?
>
> Thanks.
>

Usually if no other ID is configured the IP is taken, bu tyou could also
set an ID with my_identifier.
Try also you IPSec without IPComp (compression_algorithm deflate;)
probably Windows doesn't like that.

cheers

Burkhard Ott scripsit:

> Usually if no other ID is configured the IP is taken, bu tyou could also
> set an ID with my_identifier.

Please do you know witch ID is sent by the Windows side?

> Try also you IPSec without IPComp (compression_algorithm deflate;)
> probably Windows doesn't like that.

Setting a compression algorithm is mandatory in racoon. If it will use or
not use IPComp is setted at kernel level, if I'm not wrong.

> cheers

Thank you for your interesting.

--
Lorenzo Vaina,
MCSA Windows Server 2003,
MCTS SQL Server 2005.
messaggi privati: http://www.vaina.it/posta.html

Am Wed, 09 Jul 2008 21:32:59 +0200 schrieb Lorenzo Vaina:

> Burkhard Ott scripsit:
>
>> Usually if no other ID is configured the IP is taken, bu tyou could also
>> set an ID with my_identifier.
>
> Please do you know witch ID is sent by the Windows side?

I assume the IP Adress, if not a ID configured.


>> Try also you IPSec without IPComp (compression_algorithm deflate;)
>> probably Windows doesn't like that.
>
> Setting a compression algorithm is mandatory in racoon. If it will use or
> not use IPComp is setted at kernel level, if I'm not wrong.

You are wrong. Thats might be the reason why windows can't read the packet
correctly.

cheers

Burkhard Ott scripsit:

>> Setting a compression algorithm is mandatory in racoon. If it will use or
>> not use IPComp is setted at kernel level, if I'm not wrong.
>
> You are wrong. Thats might be the reason why windows can't read the packet
> correctly.

racoon: ERROR: /etc/racoon/racoon.conf:50: "}" no compression algorithm at
anonymous
racoon: ERROR: fatal parse failure (1 errors)

I found a Microsoft Knowledge Base article (950826) stating "You cannot
establish an IPsec connection between a Linux operating system and a
Windows Vista operating system when you initiate the connection from the
Linux operating system", but the cause description is a swithing of AH and
ESP order. However I use only AH and I have the same issue using ESP only.

>
> cheers
Thank you.

--
Lorenzo Vaina,
MCSA Windows Server 2003,
MCTS SQL Server 2005.
messaggi privati: http://www.vaina.it/posta.html

Am Thu, 10 Jul 2008 10:23:43 +0200 schrieb Lorenzo Vaina:

> racoon: ERROR: /etc/racoon/racoon.conf:50: "}" no compression algorithm at
> anonymous
> racoon: ERROR: fatal parse failure (1 errors)

AFAIK The "compression_algorithm" option in the racoon.conf only specifies
the algorithm. Compression will not be used unless you enable it in the
SPD entry.

> I found a Microsoft Knowledge Base article (950826) stating "You cannot
> establish an IPsec connection between a Linux operating system and a
> Windows Vista operating system when you initiate the connection from the
> Linux operating system", but the cause description is a swithing of AH
> and ESP order. However I use only AH and I have the same issue using ESP
> only.

I've read the article, funny issue. Try to disable AH, esp is you payload.
My idea is when MS switch teh format to esp+ah instead of ah+esp, what
happens if there is no AH attached?
Maybe it helps.

cheers

Burkhard Ott scripsit:
>
> I've read the article, funny issue. Try to disable AH, esp is you payload.
> My idea is when MS switch teh format to esp+ah instead of ah+esp, what
> happens if there is no AH attached?
> Maybe it helps.
>
> cheers

I confirm that my problem is only with the Windows Server 2008 operating
system. If the Windows side is an older Microsoft system, all works as
expected.
I tried the configuration described in the Microsoft KB article and I found
that the communication is impossible in both way, not only if Linux is the
initiator. However I need only AH and I tried using only ESP without luck:
in every cases only Windows can initiate the connection.
Now I think this is a bug in the Microsoft operating system and I'm going to
write this result on the Microsoft Winserver group.
Thank you for your help. Please continue writing here if you (or somebody
else) have some other ideas.

--
Lorenzo Vaina,
MCSA Windows Server 2003,
MCTS SQL Server 2005.
messaggi privati: http://www.vaina.it/posta.html