• February 23, 2025, 12:42:37 AM
  • Welcome, Guest
Please login or register.

Login with username, password and session length
Advanced search  

News:

This Forum Beta is ONLY for registered owners of D-Link products in the USA for which we have created boards at this time.

Author Topic: DEFRAG WITH FIRMWARE 1.03?  (Read 9633 times)

jimmay3

  • Level 1 Member
  • *
  • Posts: 13
    • MAYPRINTING.COM
DEFRAG WITH FIRMWARE 1.03?
« on: January 03, 2008, 02:21:51 PM »

Is this option gone or in the software for the PC.  I saw it on the options prior to upgrading the firmware, but after upgrade it's gone?  I have never loaded the PC software, is it there or?
Logged

chuckv

  • Level 2 Member
  • **
  • Posts: 88
Re: DEFRAG WITH FIRMWARE 1.03?
« Reply #1 on: January 03, 2008, 03:37:18 PM »

this option was removed from the firmware
Logged

jimmay3

  • Level 1 Member
  • *
  • Posts: 13
    • MAYPRINTING.COM
Re: DEFRAG WITH FIRMWARE 1.03?
« Reply #2 on: January 03, 2008, 04:01:19 PM »

Is there some other way to accomplish a defrag?
Logged

Rara Avis

  • Imperator
  • Level 2 Member
  • **
  • Posts: 76
Re: DEFRAG WITH FIRMWARE 1.03?
« Reply #3 on: January 03, 2008, 04:19:52 PM »

Ok my time has come, you will all get the filesystem rant!  The below is written as a general guide and anyone who knows better than me is invited to chime in.  The reason this applies is because the DNS-323 uses EXT*.

This is going to take a while so I will spoil the end for everyone.

EXT* does fragment, no it does not need regular defragmentation.  And there are filesystem tools, you just can't access them.


A file system is the logical addressing of physical partitions on a storage media.

Windows users will no doubt be familiar with FAT, VFAT, FAT32, and NTFS

Mac users will know and love HPFS and HPFS+

*nix heads however can not agree on anything at all, 2 families of file systems have reigned in as the default filesystem for most Linux distros, however a list of files systems available in Linux would include the following

EXT2, EXT3, ReiserFS, JFS, XFS, NFS, and many many more.

But what does that mean to you you ask?

First and foremost it means that we need to drop pre-conceived notions of what constitutes good filesystem management.  We have to accept we are not in Kansas anymore.


What exactly is fragmentation?

When a filesystem writes a file to the disk it has the very fun duty of maintaining a method of addressing it both logically and physically.  This physical addressing should be absolutely invisible to the user, the logical addressing is the file structure you are so used to.  The physical addressing is however where things get tricky.  Please see below.

For this example we are going to have a 20 character storage medium and 3 files of varying lengths, one is all asterisks, one is all pounds, and one is all zeros.  Dashes represent free space.

We write a file to the disk consisting of  8 asterisks.
********------------

We write a file consisting of 10 pounds.
********##########--

We delete the asterisk file.
--------##########--

We add back a file of 3 asterisks.
***-----##########--

We add a file of 7 zeros, but wait it won't fit sequentially so we fragment it.  We don't delete and re-write the file of 10 pounds as that would add a lot of wear and tear on the drive and take forever.

***00000##########00

As you can see now we have a system where sequential files are merely accidents after you write your first set of files equal to the capacity of the drive.


The above is an extreme simplification but it provides an idea of the process that is happening and what fragmentation is.

Now is when we get into why file systems are different.

FAT maintains a simple table that has to be parsed through as an array to perform an file based operation.  It will mercilessly drop files into the first available hole regardless of size.  In fact to even tell how much free space is available requires surveying the entire disk.

HPFS and it's childer maintains a bitmap of free space that allows it to quickly scan this bitmap to look for a large enough opening preventing most fragmentation.

Most *nix file systems (notably EXT*) do a complete third option and actually maintain a list of free space, this operation is tightly written into how files are allocated and found and is generally agreed to be a better algorithm than the FAT method.

What this means to you is that fragmentation is a comparatively rare and preventable situation on any EXT* partition.  If you do not come to near capacity and stay there then you should never notice it.  Now to get into why you can't defragment EXT3.

EXT2 can be defragmented, there is a tool, it almost never gets used in practice as it is usually unnecessary as shutting down your system to fix so small a problem never made sense.

EXT2 is extremely susceptible to a dirty shutdown and anyone who values their files and not take hours to have them hopefully recovered should always take down a EXT2 file system cleanly.

The difference between EXT2 and EXT3 is that EXT3 maintains a file system journal that aids in the recovery of the drive after an unclean shutdown.

This journal however useful however means that defragmenting the drive would require writing a new journal for every change and no one has ever (to my knowledge) seen fit to waste their time and ours on writing something to fix what is almost universally accepted as a non-issue.

You can defragment EXT3 in only one way to my knowledge, by stripping the journal making it an EXT2 file system and running the EXT2 defragmentation utility and then finally generating a new journal returning it to EXT3 status.

Now on to the infamous file system integrity check.  This side is much simpler, there is a tool, you just don't need to run it.

The file system maintenance utilities commonly used in *nix environments are old and well respected tools, they will automatically do an integrity scan every x (34 is a common number) mounts of the partition.  It is also run any time the the file system is unmounted uncleanly (read someone pulled the plug).  This can not normally be prevented and can make 1 in x  mounts take a much longer time.

This is in comparison to FAT utilities which are commonly believed to need to be run never and lead to end-user driven runs.  I could again get into the difference in the methods of the 2 primary file systems we are comparing here but I will beg you to take my word that a correctly used EXT* partition also generates less file system errors which are easier to clean.

So why not give the end user the tools (I must admit the following is an assumption of mine)?  Because they don't need them and will only generate downtime for themselves (as you can not check a mounted drive).

How do you use it correctly you ask?  Always wait for a clean take down. ALWAYS!!!

So if you insist on a file system check from a DNS-323 just pull the plug and turn it on, but be prepared to wait for it to finish and be prepared to possibly destroy your data trying to protect it.

In summary yes, EXT* does fragment, no it does not need regular defragmentation.  And there are filesystem tools, you just can't access them.
Logged
Nullum magnum ingenium sine mixtura dementiae fuit. - Seneca
There has never been a great genius without a element of madness.

chuckv

  • Level 2 Member
  • **
  • Posts: 88
Re: DEFRAG WITH FIRMWARE 1.03?
« Reply #4 on: January 03, 2008, 04:58:17 PM »

wow, that was a lot more reading than i have expected. but knowledge is great

cheers for that
Logged

mig

  • Level 3 Member
  • ***
  • Posts: 217
Re: DEFRAG WITH FIRMWARE 1.03?
« Reply #5 on: January 04, 2008, 01:00:05 AM »

Rara, great explanation!!

EXT2 is extremely susceptible to a dirty shutdown and anyone who values their files and not take hours to have them hopefully recovered should always take down a EXT2 file system cleanly.

The difference between EXT2 and EXT3 is that EXT3 maintains a file system journal that aids in the recovery of the drive after an unclean shutdown.

D-Link, have to make implementation of some kind of Un-interruptible Power Supply support...
( nut, apcupsd ) and journaling file system (EXT3) a higher priority for your firmware development!! 

In this post http://forums.dlink.com/index.php?topic=893.0
The DNS-323 needs to be powered on manually. This is to protect data in case of rolling blackouts type instances to keep the device from being powered off improperly many times.

ECF states auto power on support is intentionally disabled to protect from rolling blackouts;
however, without UPS support or Journaling file system the DNS-323 NAS has a very real
chance of data corruption after being powered off improperly just the first time.

I really like the DNS-323. I think it's a nice piece of hardware: small, low power, quiet, fast, dual SATAII,
but I really dislike the fact that the only thing that protects my data from a power outage is my own
diligent backups.

P.S. At least one of your competitors understands the importance of UPS support and a
journaling file system and has implemented these features for their recently announced dual
drive (similarly priced) NAS product.
Logged

Rara Avis

  • Imperator
  • Level 2 Member
  • **
  • Posts: 76
Re: DEFRAG WITH FIRMWARE 1.03?
« Reply #6 on: January 04, 2008, 09:27:11 AM »

Actually ECF has the right idea, again this is based on what I have been able to glean and is always subject to correction

Due to the multi-stage write nature of the EXT* family a straight power outage with EXT2 just causes a looooong recovery time, however unless you have a secondary failure (dying sectors yet undetected, blinking power during the recovery process, or a yeti) you should be at no significant risk for corruption (beyond what you get by having a spinning drive in the first place).

I have never personally seen an EXT2 drive corrupted (that wasn't automatically recovered when re-mounted) by pulling the plug, I have seen a 8 hr recovery time, on a 40 Gb drive running EXT2 (due to the frustrations that caused me to research used in the above post originally) in the middle or a shrink operation (so most of the drive was mobile).  The recovery in EXT3 should have been measured in minutes.  Please note however that I didn't lose anything but time and as soon as I dropped the drive in a trusted machine and mounted it and did my 8 hours of waiting I got everything back.  I still use that partition to this day (about a year later) and while it now has a journal I have never seen corruption on it despite a lot of abuse (it's my dedicated testing box).  On a side note, when shrinking drives you want to use your most reliable hardware and a freshly burned perfect boot CD, failure to do so will cause an embarrassing recovery process.

The big risk with the DNS-323 is that there will be a power outage and someone will get home turn it on wait 15 minutes (on the outside) and then decide it has crashed and needs power cycled because they don't know how long that recovery can take and have no indication that is what the device is doing.  This WILL cause copious data loss.

in short you guys are right we need journaling and UPS management, what's the hold up ECF?

P.S.  I'm glad I could help guys!
Logged
Nullum magnum ingenium sine mixtura dementiae fuit. - Seneca
There has never been a great genius without a element of madness.

fordem

  • Level 10 Member
  • *****
  • Posts: 2168
Re: DEFRAG WITH FIRMWARE 1.03?
« Reply #7 on: January 04, 2008, 06:15:48 PM »

Ok my time has come, you will all get the filesystem rant!  The below is written as a general guide and anyone who knows better than me is invited to chime in.  The reason this applies is because the DNS-323 uses EXT*.

This is going to take a while so I will spoil the end for everyone.

EXT* does fragment, no it does not need regular defragmentation.  And there are filesystem tools, you just can't access them.


No - I'm not going to disagree with you - in fact I have presented similar arguements on other fora and had other users present the well expounded theory that - take your choice - fragmentation does not occur/defragmentation is not required because it's EXT*

As an old disk service tech - dating back to the days of 14", 5MB per platter disks that we routinely replaced heads and platters on - and manipulated sector by sector - the simple facts are that DAS - direct access storage is still accessed exactly as it was in the late '70's.

Fragmentation occured then and fragmentation occurs now - it may be less of a problem with EXT*, but exactly how much less is dependent on how much free space is available.
Logged
RAID1 is for disk redundancy - NOT data backup - don't confuse the two.

Rara Avis

  • Imperator
  • Level 2 Member
  • **
  • Posts: 76
Re: DEFRAG WITH FIRMWARE 1.03?
« Reply #8 on: January 06, 2008, 10:57:34 PM »

You are absolutely right, my argument wasn't that is does or doesn't fragment, but rather that the minds who designed the thing didn't consider the problem bad enough to design tools in the first place so I am not concerned.  Additionally it's good to see some old skill still haunting the fora, I look forward to seeing you around.
Logged
Nullum magnum ingenium sine mixtura dementiae fuit. - Seneca
There has never been a great genius without a element of madness.