Following are the list of items that I always follow after installing a new Drupal site. Please put some comments if I need to add some other points here or if you are doing it differently.

1. Install Administration menu module.

Administration menu module provides an easy and time saving way to navigate to your site. This is a very helpful module to all drupal developers and administrators. The module renders all administrative menu items below 'administer' in a clean, attractive and purely CSS-based menu at the top of your website.

a. Download admin_menu module from http://drupal.org/project/admin_menu
b. Unzipped it in your directory for contributed module.
c. Go to Site Building -> Modules -> List and install Administration menu module.

2. Enable/Disable core modules.

Enable necessary core modules based on your site requirements. The modules that you can install in every type of site are contact, search, path and upload. Also there are some Unnecessary Drupal modules which we don't want in every site that we built. For example color module.

So, go to Site Building -> Modules -> List. Check/Uncheck the modules based on your requirement and save the configuration.

3. Add/edit the site informations

a. Go to Site Configuration -> Site Information
b. Add or edit the information about your newly created site.

4. Edit the public registration settings

Edit the public registration settings of your site so that administrator approval is required on user registration.

a. Go to User Management -> User Settings.
b. Select "Visitors can create accounts and no administrator approval is required."
c. Save the configuration.

5. Add User role

If you want your site to be maintained by other users also, then you should create some user role and configure access permission.

a. Go to User Management -> Roles
b. Add Roles e.g., editor, Administrator, member
c. After adding the roles, configure role permission clicking next to each role.

6. Install some necessary contributed module

Install some necessary contributed module based on your requirements. Following are the modules that are very useful and you can use in your every site.
a. Pathauto (http://drupal.org/project/pathauto)
c. cck (http://drupal.org/project/cck)
d. views (http://drupal.org/project/views)
f. devel (http://drupal.org/project/devel)
g. backup_migrate (http://drupal.org/project/backup_migrate)
h. wysiwyg editor (http://drupal.org/project/wysiwyg)
i. Search 404 (http://drupal.org/project/search404)

7. configure cron.

8. Configure Search

Search is one of the main features of every site. So configure it properly. Go to Site configuration -> Search settings and change your settings.

9. Configure site date/time

10. Add some content.

Now, since the site is ready for more advance developement, you can add some content for testing.