Oracle Direct NFS configuration using Linux or Windows with 11g and 12c

Starting with Database Version 11g, Oracle offers with Direct NFS an integrated NFS-Client, which is independent from the underlying OS-based NFS-Client. This allows direct access to the NFS storage for the Oracle Instance and realizes Oracle’s own optimizations. In addition, one can implement multipathing over four parallel ports, independently from any switches, NIC bonding or teaming. The Oracle Instance configures the communication with the NFS automatically and thus facilitates the configuration under RAC. Using Oracle Direct NFS, shortly dNFS, also provides using NFS Storage with Oracle Instances on Windows systems.

The Direct NFS Client supports NFS Server using NFSv3 with database version 11g and from 12c on NFSv3 to NFSv4.1. Before Direct NFS can be used, it must be enabled and set up at least for a Single Instance. For that two requirements need to be fullfilled. First the Direct NFS ODM (Oracle Disk Manager) Library must be loaded at the strat of the Instance and second the mount point and NFS storage needed to be configured. Following the configuration tasks with Linux and Windows:

1. Loading Direct NFS ODM Library

Using Linux the Direct NFS Client had to be activated.

Linux: Login as Oracle software installation owner

> cd $ORACLE_HOME/rdbms/lib
> make -f ins_rdbms.mk dnfs_on

This option is only supported since 11.2.

Manually, you can do also the following:

> cd $ORACLE_HOME/lib
> mv libodm11.so libodm11.so.stub
> ln -s libnfsodm11.so libodm11.so

Using Windows the standard ODM Library, oraodm11.dll had to be replaced by ODM NFS Library, oranfsodm11.dll in the bin Directory.

Windows: Shutdown the database and change into ORACLE_HOME\bin directory

> copy oraodm11.dll oraodm11.dll.stub
> copy /Y oranfsodm11.dll oraodm11.dll

At the start of the database following lines should be found in the Alert Log for both Linux and Windows:

Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0

2. Configurating mount point and NFS storage

a. Mount point configuration

Linux:

To create the mount points using Linux in general it is sufficient to configure the /etc/fstab or /etc/mtab. In addition the oranfstab can be created to use Oracle specific options. The Direct NFS Client determines the settings of the mount point(s) in succession and if available

1. $ORACLE_HOME/dbs/oranfstab
2. /etc/oranfstab
3. /etc/mtab

Linux: fstab mount options – You should give “deafult” a chance. Are there any problems the key entry is the NFS buffer size parameters rsize and wsize. Oracle recommends a value of 32768. Using RAC the entry in the fstab should be the same on every node.

nas1:/volume1/oranfs    /mnt/oranfs nfs     rw,bg,hard,rsize=32768,wsize=32768,nfsvers=3,nointr,timeo=600,actimeo=0,tcp,addr=172.16.0.2     0 0

Linux: Optional oranfstab (minimum information bold)

server: nas1             # alias freely selectable
path: 172.16.0.2# IP NFS server
local: 172.16.30.40       # optional IP database server
path:                     # optional up to 4 path/local pairs for multipathing
local:                    # optional multipathing for load balancing and failover
export/volume1/oranfs/oradata mount: /mnt/oranfs/oradata
                          # export: NFS server path, mount: local path
export: mount:            # optional additional export/mount paths - unlimited
mnt_timeout: 30           # optional timeout when mounting fails
dontroute                 # optional messages should not be routed by OS
nfs_version: nfsv4.1      # optional from 12c + NFSv4 (default: NFSv3)
management                # optional from 12c use management interface for SNMP
community                 # optional from 12c community string for SNMP

server: nas2              # additional server seperated by blank line
path: 172.16.0.3
export: /volume/path/nfs mount: /local/path

Windows:

In opposite to Linux the oranfstab has to be used with Windows. The Direct NFS Client determines the mount options only using ORACLE_HOME\dbs\oranfstab. Accordingly the oranfstab file has to be created manually (as textfile without file extension). The specified mount path (here: D:\oranfs\oradata) hasn’t to be exisit on the Oracle-Server. If you specify the path the Oracle Instance automatically detects that it is a NFS path and puts the file on the NFS server. Using RAC the oranfstab has to be created identically on all nodes.

Windows: oranfstab (minimum information bold)

server: nas1              # alias freely selectable
>path: 172.16.0.2          # IP NFS server
local: 172.16.30.40       # optional IP database server
path:                     # optional up to 4 path/local pairs for multipathing
local:                    # optional multipathing for load balancing and failover
export: /volume1/oranfs/oradata mount: D:\oranfs\oradata
# export: NFS server path, mount: local path
export: mount:            # optional additional export/mount paths - unlimited
mnt_timeout: 30           # optional timeout when mounting fails
uid: 65534                # optional User ID for NFS server access 
gid: 65534                # optional Group ID for NFS server access
nfs_version: nfsv4.1      # optional from 12c + NFSv4 (default: NFSv3)
server: nas2              # additional server seperated by blank line
path: 172.16.0.3
export: /volume/path/nfs mount: D:\local\path

b. NFS storage configuration

The NFS server must at least support NFS version 3.

Espacially with Linux-based servers, there is a problem with port reservation, because the Oracle Instance in not running as root user. It is therefor necessary to prevent rejection by the NFS server on this by setting the insecure option in the /etc/exports. This allows connections to non-priviliged ports above 1024.

nas1> cat /etc/exports
 
/volume1/oranfs *.carajandb.intra(rw,async,no_wdelay,insecure,root_squash,insecure_locks,sec=sys,anonuid=1024,anongid=100)

To ensure, creating a tablespace on the NFS server works correctly, the permission should be set correctly.

Linux:

nas1> cd /volume1/oranfs
 
> chown oracle:dba oradata
> chmod 755 oradata

Windows: The Direct NFS Client uses by default ui:65534 and gid:65534, which should correspond with nobody and nogroup on the NFS server. The access should be ensured but can in doubt be set in the oranfstab. For the right permissions you can create an user oracle and a group ORA_DBA on the NFS server and set the permissions accordingly:

nas1> cd /volume1/oranfs
 
> chown oracle:ORA_DBA oradata
> chmod 755 oradata

c. Checks

First a query to check if the NFS server is connected:

select * from v$dnfs_servers; 
 
D  SVRNAME      DIRNAME                    MNTPORT  NFSPORT  WTMAX    RTMAX
--  ------------ -------------------------  -------  -------  ------   ------
1   nas1          /volume1/oranfs/oradata   892      2049     131072   131072

If the configuration is not correct, the query simply returns no rows. If the instance already was restarted it is also possible that the path must addressed first before the query returns a result. Create a file on the mount for testing and repeat the query:

Windows:

SQL> create tablespace test_tbs datafile 'D:\oranfs\oradata\test_tbs.dbf' size 10M;

Linux:

SQL> create tablespace test_tbs datafile '/mnt/oranfs/oradata/test_tbs.dbf' size 10M;

If the configuration runs, you can determine if the access to the NFS really runs over Direct NFS by using the following query. Returns the query no rows the NFS-Client of the operating system is used at Linux. Help is only given by the Alert Log und Trace files.

select * from v$dnfs_files; 
 
FILENAME                          FILESIZE          PNUM     SVR_ID
--------                          --------         ----     ------ 
d:/oranfs/oradata/test_tbs.dbf    10493952         10       1

More information can be retrieved with following views:

select * from v$dnfs_channels;
select * from v$dnfs_stats;

d. Troubleshooting

Linux: A possible source for problems could be the wrong permission on oradism. The Direct NFS Client needs root privileges for initializing the communication with the NFS storage. In this case the owner is not root but oracle. The Alert Log shows up with the following error message:

Direct NFS: please check that oradism is setuid
Direct NFS: warn could not mount /volume1/oranfs/oradata on svr 172.16.0.2 via dNFS

> ls -l $ORACLE_HOME/bin/oradism
-rwxr-x--- 1 oracle oinstall
> chown root:root oradism
> chmod 4755 oradism

Generally the permissions look like the following:

> ls -l $ORACLE_HOME/bin/oradism
-rwsr-x--- 1 root oinstall