Cisco 20x20 cards

[TheGreyRider]

Anyone using the new MC20x20 cards for Cisco 10K's?

[jbarbieri]

No, not yet.

My new CMTS is "supposed" to be coming in with them. But that's not going to be for another 4 months apparently.

[buzzwork]

4 month ? BU on the mc20x20 said 70 days here

[jbarbieri]

4 month ? BU on the mc20x20 said 70 days here

I guess being a smaller MSO, they can push ours out.

But yea, they told us July for our new chassis (SIP600, PRE4, 20x20)

[TheGreyRider]

I was hoping someone out there was using them to see if I am the only one with so many problems and issues. From what i am gathering from Cisco we are the first with these in production and now i see why no one else has been in a hurry to get these out in the field.

[slinenbe]

I have the 20x20 cards in production with roughly 4400 subscribers. I've found the biggest issue I've faced so far is SB3100 and SB4100 cable modems with BPI+. To work around the issue I had to downgrade the SB3100 and SB4100 modems to the last 1.0 firmware. Opening a ticket with TAC for the issue, I had to FedEx sample modems and and was told that the RSA database for the 20x20 cards didn't accommodate the SB3100 and SB4100 modems with cert sizes greater than 64 Bytes (big surprise ), which the 5x20 support without issue. So, it looks like my wide scale deployment has been slowed down considerably since I'm stuck in developer land for a true fix, which may or may not happen.

After letting everything burn in, I'm also seeing some other issues crop up with DLINK DCM-202s and am waiting on samples from my field personnel. What is everyone else facing with the 20x20 cards?

Thanks

[TheGreyRider]

We had issues with BPI as well, that was a Cisco BUg that was fixed in SCC2, that version had another BPI issue that the work around was modifications to my LB groups, that is to be fixed in SCC3 which should be out very soon. We had a lot of issues when we pulled 5x20s and inserted 20x20 cards. Long story short we are now having to reload the chassis every time we install another 20x20 card, this too i was told will be fixed in SCC3. We currently have approx 10K to 12K subs on 20x20 cards with many more to be installed in the next few months.

[slinenbe]

We're running SCC2 at the moment and it looks like we finally made it into the bug database (CSCtf67220). We're now running into cooling issues with slots 8/0 and 8/1 running the 20x20 cards, which didn't have with the 5x20 models.

[wittmann]

We got some crashes on and off on linecards. This issue should be solved on SCC3. SCC3 should be delivered end of next week. At the beginning of this week we want to try the Pre-Engineering Release from the SCC3 who should have fix the problem.

It's Bananaware. Maturation by Customer. To harvest green, going yellow by customer . And hey, where are the Certification by CableLabs or Cable Europe Labs?!?!

[aka]

yeah we just upgraded to 20x20 on our first box last night, using SCC3. No majors issues as of yet but give it time. Had to tweak voice QoS settings to accomodate for packet loss on those cpes' Also load-balancing group ids need to be unique on a per card basis. I will update with additional findings as we will be moving forward with upgrades.

[buzzwork]

Hey,

Have any of you experinced any more "bugs" in SCC3 ? Would like to hear about it before making the same mistakes

Martin

[wittmann]

We have a lot of trouble. SCC3 is on 4 fully loaded chassis and one half loaded chassis. We got in average all 4 days one reboot on different Linecard without Crashlogs and Tracebacks. The througput on Linecards going down until you will reboot the Linecard. Next debug images is anounced for next week, also Cisco will send us diag-cables direct to the linecards.

Try M-CMTS with MC520H-Linecards. With our M-CMTS with SCB4 and MC520H-Linecards we got no trouble. For EdgeQAM we use Arris D5.

[buzzwork]

We just got some mc2020 cards thats we planed on putting into our 10k today.

The reboots you talk about is reboots of the mc2020 cards right, not the chassie/10k or already running mc5x20 cards right ?

Martin

[wittmann]

We just got some mc2020 cards thats we planed on putting into our 10k today.

The reboots you talk about is reboots of the mc2020 cards right, not the chassie/10k or already running mc5x20 cards right ?

Martin

right! But we don't use mixed chassis with MC20X20V and MC520H/D/U. I can't say what happened with the MC520-LC with the SCC3. Our Chassis with CMTS and M-CMTS are have only MC520H with IOS-Version 12.2(33)SCB4.

The SCC3 has also a Bug with cable filter configured on the chassis. The Traffic from MTAs will going into CPE-Group. Workaround is to configure a access row for the MTA-Traffic into the CPE-Group.

I should me rename to Banana-Joe

[buzzwork]

wittmann, does your entire 10k reboot or just your mc2020 linecard ?

Martin

[wittmann]

wittmann, does your entire 10k reboot or just your mc2020 linecard ?

Martin

Hi Martin,

only the linecard.

best regards,

Peter

[jbarbieri]

Oh boy, sounds like I will have my work cut out for me when the 2020 cards come in.....eek

[aka]

Yes I have about 5 full chassis in our market so far and are about to upgrade 2 more this week. Have been running SCC3 for about 4 weeks now with no major probs. so far. whitman are you using PRE-4 or 2? I am using PRE4s, but some smaller markets in my region are running 20x20s on PRE2 and are having the same problem you are having with linecards resetting. I am assuming b/c 2 MC20x20Vs is the same as running a full box of MC5x20Us, this will cause processing memory problems, for this reason Cisco only suggest using 2 MC20x20s in a chassis using PRE2s, food for thought. Also make sure you format your compact flash cards in the box prior to using them, will save u some trouble. I will post more info when I get it.

-aka

[wittmann]

Hi aka,

at this time, we use only the PRE2. Thanks for your comment.

Best regards,

Peter

[buzzwork]

aka, can you comment more on the "only 2x mc2020 when using PRE2" and format compact flash card before mc2020 ??
Would VERY much like to hear even more on this PLZ

Maybe some Cisco/TAC doc.

Martin

[aka]

Buzzwork, yeah I should have explained better, with the flash disk info, the compact flash that comes with the PRE4 will not be read from/ written to properly unless it is formatted within the PRE4 first, "format disk0:", "format stdby-disk0:". As far as the 20x20s, 2 of the 20x20s will provide the same amount of downstreams as a full chassis of 5x20s, providing you are licensing your cards as true 20x20s. PRE2 were not designed to meet specs of 60 - 140 downstreams. And as a cya cisco decided to recommend that no more than (2) 20x20s be used on a chassis with PRE2. Also it is recommended that on a MCMTS with PRE2s 20x20s not even be used. Sorry I am not at liberty to release any docs on this right now as they are unofficial. I will update this post with more info when received.

~aka

[buzzwork]

hey Aka,

I have asked Cisco SE and TAC about this issue with >2 mc2020 on PRE2 and they have no knowledge about this claim.
The only issue there is, is the amount of memory pr user on SCC + PSU power consumption with fully loaded chassie.
Martin

[wittmann]

@aka,

i believe there is no problem for the PRE2 with the MC20X20V itself. I think, the important topic is are the DOCSIS MAC Management Messages (MMMs) and the summary of den MAC Domains (Interface Cable). At a fully loaded 10k chassis the count of the Interface Cable is the same as with MC520H: 5 per LC, 20 on the whole Chassis. You can see this on a show version. The MC20X20V has the same count of Interface Cable = 5. What we have is only more physical Downstreams via Controller IntegratedCable. The mapping of the MMMs to the extended Downstreams of the IntegratedCable are only on the same Linecard. You can not share one Controller IntegratedCable of other linecards. So i think the performance which you need to mapping the MMMs from the Interface Cable to the Controller IntegratedCable are handled by the power from the MC20X20. Thats my theory.

Let's see whats comming next...

Thanks for conversation.

[bfolks05]

Gents, I have some expierence with the 20x20s and the diff platforms associated with it. And I have heard the arguments presented by aka and wittman, I think at this point with this platform being generally new, the statements of using 2 or less 20x20s with the PRE2 is more of a speculation used to provide some sort of explanation for the time being for linecard failures, hence the unofficialness on the vendors part. I havent dived into this much but I can see how a box running 20x20s with PRE2 could have issues, just based a processing/cpu and memory usage at the least. Might be reading a little much into with the MMMs. aka, do u have any full 20x20 chassis with PRE2? Has anyone upgraded to SCC3+? Also what kind or load balancing are you guys using on those, and how is it going?. Thanks all

[slimjim100]

Basically the PRE-2 and only (2) 20x20's per 10k is based on if the 20x20's are fully configured and at full card/line rate as the PRE-2 throughput was designed to handle (8) 5x20's or 40DS connections and with (2) 20x20's you get 40 DS connections from just 2 line cards. So if you are running HCCP (line card protection) you do not have to count that card. Now there is nothing that says you can not have more than (2) 20x20's in a PRE-2 but if you are fully loading them you might start to bottleneck at the PRE-2. With all the 20x20 deployments I have seen to date I have seen a few with 20x20's on a PRE-2 with more than (2) cards per box but the customer was not useing all 4 DS channels per port so it was not a worry. I am not sure if there is any official documents on this but if you get the spec sheets for the PRE-2 you can do the math.

Cheers,

Brian

(the above statement is my own opinion and claim)

[kryhavoc]

I work for a slightly larger MSO... we have PRE4 10k's fully loaded with 20x20 cards and they can easily support up to about 20k subs per box with 4x4 service groups. We are running SCC3 at the moment with a ton of DCC/UCC utilization based load balancing keeping most everything even on channel usage. Reboots are scary now.

[jbarbieri]

We have our new CMTS with PRE-4s and 4+1 20x20 cards running, has been for about 2 weeks in production now. Running SCC3.

:knock on wood: so far so good.

Hit a couple of bugs on another 10k running SCC3 with 520 and wideband SPAs. First one was we had a failover, and now the load-balance groups won't go into an up state when using method utilization, but using method modems brings them up. TAC is trying to figure out why.

Another is that we are now having random voice problems. Seemed to only be associated with 2 US ports in the same MAC domain. The MTA IP would be non-responsive to pings, and the eMTA would show it was stuck trying to get the provisioning file from the TFTP server. Doing a clear ip arp <ip address> fixes this problem. Got to a point where we did a clear arp-cache on the whole box, and has been running with no problems for about a week now.

Other than those problems, nothing else major *yet*. Those 20x20 cards are pretty sweet. The users on that CMTS are now happy (myself included, I was one of the unfortunate souls on the CMTS that it replaced). No more speed issues for now!

[buzzwork]

We been running production some months now with PRE2's and 2x mc2020 using SCC3 and no problems at all ... all runs perfectly with some help from SlimJim100 on the config

[jbarbieri]

Anyone running into any:

load-balancing problems?
VoIP problems?
ARP problems?


We have been getting hit with all 3:

When using method utilization, the groups go into a down state.
Random eMTAs lose dialtone.
Problems with DCC and CPEs that have same MAC/different IP addresses.

[davidj]

We are running PRE4 with 20 x 20 cards. IOS is 12.2 (SCC3). Issue we are seeing are random customers loosing throughput. It looks to have been tracked down to ARP issues on the uBR10k, whereby customers CPE's are ARPing for a gateway and the CMTS is not sending a response.

Also just experienced our first outages:
1. CPUHOG and Traceback entries in syslog at the time one of our uBR10k's lost it's marbles!
2. Card in slot5/0 just dropped its modems. No syslog entries or anything obvious. Card appeared to be working so failover didn't work! Forced switch to protection restored cable modems. (Cisco have RMA'd the card.)

[buzzwork]

[quote="davidj"]We are running PRE4 with 20 x 20 cards. IOS is 12.2 (SCC3). Issue we are seeing are random customers loosing throughput. It looks to have been tracked down to ARP issues on the uBR10k, whereby customers CPE's are ARPing for a gateway and the CMTS is not sending a response.

Have you opend an tac case and have the developers found the problem? maybe an bug id you can post? Would be nice to hear more about these problems

Martin

[davidj]

TAC case is open and the Cisco developers have found an issue. To date, we haven't heard anything official on a course of action.

[mpb]

hey david,

would be useful if you could share bug id for that.

so far our 20x20 have been ok with SCC3 and SCC4, but seen some issues with the WB-SPAs


thanks

[docsisgeek]

Our 20x20 cards have been fairly interesting so far while running SCC2. I plan on upgrading to SCC4 to get away from the non-working "show cable modem offline" per interface commands. That is such an annoying bug while you're trying to troubleshoot a chassis.

So far we've had these problems:

1) One Ironbus crash the other day which wiped out all of the modems. This was fixed automatically after the linecards rebooted but the customers on the chassis were not too happy.
2) We had our downstreams configured to use 100% of the downstream which was causing intermittent flaps for modems on those downstreams. I changed the max downstream usage to 96% to allow for proper DOCSIS maintenance and now we seem fine.
3) We also have a new issue that has cropped up where devices that change IP address behind a cable modem tend to be included in the ARP table, but will not show up behind a cable modem. This does not allow the CPE to transmit or receive data. Cisco TAC seems confused as to how this could happen so I think I'm just going upgrade and see if the problem goes away with the upgrade...

Cheers.

-James

[davidj]

RE: Arp issue.....We are upgrading to 12.2(SCC5) in the next week or so. This "should" fix the problem

Also, one 20x20 linecard rebooted 6 times over a 36 hour period. Luckily, it was card 5/1.