Hopper + HWS and the 3TB limit (Tech discussion)

[John Kotches]

I've been working on a move from 2TB to 3.0TB drive over the weekend and thought I'd discuss this topic. Now that I've "walked a mile in their shoes" I understand why drives are limited to 3TB. I don't understand why they chose to take this path to get to where they are, but that's another story for another day.
[
First off, they use MBR/MSDOS partitioning, yeah, good ole fdisk and all its limitations. So that means there is a 2TB limitation for any partition. You can have a > 2TB drive so long as the partitions don't exceed 2TB.

So here's the partition table from the 2.0TB drive I started with:

Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes
1 heads, 63 sectors/track, 62016336 cylinders
Units = cylinders of 63 * 512 = 32256 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

 Device   Boot  Start  End     Blocks  Id  System
/dev/sdc1         2       33290  1048576  83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sdc2  33290  16677353  524288000  83  Linux
Partition 2 does not end on cylinder boundary.
/dev/sdc3  16677353  33321417  524288000  83  Linux
Partition 3 does not end on cylinder boundary.
/dev/sdc4  33321417  62016336  903889974+  85  Linux extended
Partition 4 does not end on cylinder boundary.
/dev/sdc5  33321418  49965481  524288000  83  Linux
/dev/sdc6  49965482  62016336  379601911  83  Linux
[

They've taken this drive and split into an odd configuration. There's a very small partition (sdc1, about 1GB) that is empty. I'm not sure what role it plays in things. Next, they build 2x 500GB partitions (sdc2+sdc3 @ 524280000) you'lll have to trust me that it comes out to 500GB in binary math.

This next part is where it gets funky... to avoid running out of primary partitions (sdc1+2+3 are primary partitions) they build partition 4 with the remainder of the disk as an extended partition. This is inherently limited to 2TB because we're using fdisk. That's reason 1 for the 3TB disk limit.

We still have unused space on the disk, because they keep carving up the disk within the extended partition. So, we have a 500GB partition (sdc5) and a ~375GB (sdc6) partition within the extended partition.

When you move over to a 3TB drive, you run into the next limitation, which is no more than 8 partitions on an MBR/MSDOS partitioned disk. They simply extend the partitioning scheme outward thusly:

Disk /dev/sda: 3000.6 GB, 3000592982016 bytes
1 heads, 63 sectors/track, 93024336 cylinders
Units = cylinders of 63 * 512 = 32256 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

  Device Boot  Start  End  Blocks  Id  System
/dev/sda1  33  33321  1048576  83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2  33321  16677385  524288000  83  Linux
Partition 2 does not end on cylinder boundary.
/dev/sda3  16677385  33321448  524288000  83  Linux
Partition 3 does not end on cylinder boundary.
/dev/sda4  33321448  93024336  1880640984  85  Linux extended
Partition 4 does not end on cylinder boundary.
/dev/sda5  33321481  49965544  524288000  83  Linux
/dev/sda6  49965577  66609640  524288000  83  Linux
/dev/sda7  66609673  83253736  524288000  83  Linux
/dev/sda8  83253737  93024336  307773880+  83  Linux

So you can see that this disk got extended out to 8 partitions (sda 7+8), with 1,2,3,5,6 and 7at 500GB and the 8at around 300GB.

I understand what Dish has done, but at this point I haven't the faintest clue as to why this method was selected.

Please bear with me, I'll probably be editing this several times to get the spacing just so in the code sections.

 

[charlesrshell]

I've been working on a move from 2TB to 3.0TB drive over the weekend and thought I'd discuss this topic. Now that I've "walked a mile in their shoes" I understand why drives are limited to 3TB. I don't understand why they chose to take this path to get to where they are, but that's another story for another day.

John, I am not too knowledgeable about hard drives. Are you working with the Dish Hopper or external hard drive? Doesn't the Hopper have a 2TB drive and Dish wants a limit of 2TB external hard drive size?

 

[John Kotches]

I'm working with externals only.

Sent from my Nexus 6 using Tapatalk

 

[Tron2012]

First (smaller) partition may be be used for data swapping purposes, to supplement RAM.

 

[charlesrshell]

I'm working with externals only.

Sent from my Nexus 6 using Tapatalk

Thanks John. Good luck to you. Way above my level.

 

[Tron2012]

Dish probably never really puts a priority on external storage concerns. They make it work for EHD's and that's it. Thinking in their Linux-based OS they just carried over the base settings from generation to generation and never enabled GPT for larger partitions since their internal HDD's are not monster sized. Yet.

 

[ChadT41]

Dish probably never really puts a priority on external storage concerns. They make it work for EHD's and that's it. Thinking in their Linux-based OS they just carried over the base settings from generation to generation and never enabled GPT for larger partitions since their internal HDD's are not monster sized. Yet.

I agree but they did recently in the last 24 months start supporting 2tb compared to previously, 1tb for EHd.

 

[Tron2012]

I agree but they did recently in the last 24 months start supporting 2tb compared to previously, 1tb for EHd.

Correct. But still not past the standard 2TB limit for partitioning and not generally considered "monster drives"! Can't see them making a move soon to much larger drives either. Why they support the EHD option. Let the customer bear the cost and headache of managing thousands of recordings.

I know and respect it's different strokes for different folks, but never understood why people have a bunch of EHD's and store massive amounts of recordings. Especially if they ever move away from Dish then all the recordings become useless. Understand the movie library idea but man, that's a lot of TV time! Barely can keep up with regular programming let alone watching movies I've seen before! No haters, please! Just commentary from my perspective.

 

[navychop]

This just makes me want to get back to my Linux project to catalog my EHDs. I just don't think it would be hard, but I -REALLY- don't have the time. Just finishing up our tile laundry room floor.

THIS project, sad to say, isn't even in volume 1 of the To Do List anymore. It's a "me" item, not an "us" item.

BUT- I may just spring for a 3 TB hardware RAID and be done with it.

 

[charlesrshell]

This just makes me want to get back to my Linux project to catalog my EHDs. I just don't think it would be hard, but I -REALLY- don't have the time. Just finishing up our tile laundry room floor.

THIS project, sad to say, isn't even in volume 1 of the To Do List anymore. It's a "me" item, not an "us" item.

BUT- I may just spring for a 3 TB hardware RAID and be done with it.

Priority issues!!

 

[TheKrell]

I always thought that the 2GB partition included the decryption keys for the account. I'll bet it isn't really zero-filled. <-- my wag

 

[John Kotches]

I haven't gone hunting for keys... Yet. I'm just moving data around.

The 2TB partition isn't directly accessed. It is the extended partition that contains the additional space beyond the first TB. The space is accessed through logical partitions 5-8. As many as are needed to use all the drive.

I suspect that the keys might be in the 1GB partition (partition 1) but I have not investigated it.



Sent from my Nexus 6 using Tapatalk

 

[John Kotches]

Thinking out loud here...

The keys don't need to be on the external at all. The encryption could be upstream of the write. The drive would just be a bucket of bits.

Most efficient would be a cryptochip/ASIC or an FPU to do encryption chores. The datastreams aren't particularly intensive (low data rate) so even multiple streams would be possible easily.

This also means the keys to the store remain inside the Hopper / HWS. It would be more secure this way.

Sent from my Nexus 6 using Tapatalk

 

[Tron2012]

Good luck! I'll give Dish credit and suspect that in order to allow EHD transfers/storage of electronic assets they do not own outright (recordings) that they had to fully demonstrate to the movie and TV moguls that their encryption standards and protocols were as protected from prying eyes as much as possible. So that the movement of recordings was secure and not easily (if at all) hacked. To John's point above.

Still say the 1GB is for swapping and nothing of interest or secretive. Why it is most likely empty and not a big partition.