How to Read Hdf in Python by Certain Incidies
Read and utilize some unlike file formats¶
In this session, we will learn to employ some geospatial tools using GDAL in Python. A good set of working notes on how to employ GDAL has been developed that y'all will find useful for groundwork reading.
HDF(Hierarchical Information Format) and HDF-EOS are common formats for EO data and so you need to have some idea how to use and dispense them.
A hierarchical data format is essentially a format that 'packs' together diverse aspects of a dataset (metadata, raster information etc.) into a binary file. In that location are many tools for manipulating and reading HDF in python, merely we will use ane of the more than generic tools, gdal here.
When using HDF files, we need to accept some idea of the stucture of the contents, although you can clearly explore that yourself in an interactive session. MODIS products have extensive data available to help you lot translate the datasets, for case the MODIS LAI/fAPAR production MOD15A2. We will use this as an example to explore a dataset.
You volition need access to the file `MCD15A2.A2011185.h09v05.005.2011213154534.hdf <http://www2.geog.ucl.ac.britain/~plewis/geogg122/_images/MCD15A2.A2011185.h09v05.005.2011213154534.hdf>`__, which you might access from the MODIS Land Products site
Before going into the Python coding for GDAL, information technology is worthwhile looking over some of the tools that are provided with GDAL and that tin can be run from the trounce. In particular, nosotros tin apply the gdalinfo program, that takes a filename and volition output a copious description of the data, including metadata, but as well geogrpahic project, size, number of bands, etc.
!gdalinfo MCD15A2.A2011185.h09v05.005.2011213154534.hdf Driver: HDF4/Hierarchical Data Format Release iv Files: MCD15A2.A2011185.h09v05.005.2011213154534.hdf Size is 512, 512 Coordinate Organisation is `' Metadata: HDFEOSVersion=HDFEOS_V2.9 LOCALGRANULEID=MCD15A2.A2011185.h09v05.005.2011213154534.hdf PRODUCTIONDATETIME=2011-08-01T15:45:34.000Z DAYNIGHTFLAG=Day REPROCESSINGACTUAL=reprocessed LOCALVERSIONID=5.0.4 REPROCESSINGPLANNED=further update is anticipated SCIENCEQUALITYFLAG=Not Investigated AUTOMATICQUALITYFLAGEXPLANATION=No automatic quality assessment is performed in the PGE AUTOMATICQUALITYFLAG=Passed SCIENCEQUALITYFLAGEXPLANATION=See http://landweb.nascom/nasa.gov/cgi-bin/QA_WWW/qaFlagPage.cgi?sat=aqua the product Science Quality status. QAPERCENTMISSINGDATA=3 QAPERCENTOUTOFBOUNDSDATA=3 QAPERCENTCLOUDCOVER=23 QAPERCENTINTERPOLATEDDATA=0 PARAMETERNAME=MCD15A2 VERSIONID=5 SHORTNAME=MCD15A2 INPUTPOINTER=MYD15A1.A2011192.h09v05.005.2011194222549.hdf, MYD15A1.A2011191.h09v05.005.2011194224921.hdf, MYD15A1.A2011190.h09v05.005.2011192044353.hdf, MYD15A1.A2011189.h09v05.005.2011191043320.hdf, MYD15A1.A2011188.h09v05.005.2011190043751.hdf, MYD15A1.A2011187.h09v05.005.2011189125127.hdf, MYD15A1.A2011186.h09v05.005.2011188044024.hdf, MYD15A1.A2011185.h09v05.005.2011187152544.hdf, MOD15A1.A2011192.h09v05.005.2011194044548.hdf, MOD15A1.A2011191.h09v05.005.2011193042953.hdf, MOD15A1.A2011190.h09v05.005.2011192045720.hdf, MOD15A1.A2011189.h09v05.005.2011191045425.hdf, MOD15A1.A2011188.h09v05.005.2011190045726.hdf, MOD15A1.A2011187.h09v05.005.2011189045204.hdf, MOD15A1.A2011186.h09v05.005.2011188195439.hdf, MOD15A1.A2011185.h09v05.005.2011210211523.hdf, MCD15A2_ANC_RI4.hdf GRINGPOINTLONGITUDE=-103.835851753394, -117.486656023174, -104.256722414513, -92.131858571552 GRINGPOINTLATITUDE=29.8360532722546, 39.9999999964079, 40.0742066197196, 29.9009502428382 GRINGPOINTSEQUENCENO=one, 2, 3, 4 EXCLUSIONGRINGFLAG=N RANGEENDINGDATE=2011-07-11 RANGEENDINGTIME=23:59:59 RANGEBEGINNINGDATE=2011-07-04 RANGEBEGINNINGTIME=00:00:00 PGEVERSION=5.0.9 ASSOCIATEDSENSORSHORTNAME=MODIS ASSOCIATEDPLATFORMSHORTNAME=Terra ASSOCIATEDINSTRUMENTSHORTNAME=MODIS ASSOCIATEDSENSORSHORTNAME=MODIS ASSOCIATEDPLATFORMSHORTNAME=Aqua ASSOCIATEDINSTRUMENTSHORTNAME=MODIS QAPERCENTGOODQUALITY=100 QAPERCENTOTHERQUALITY=100 HORIZONTALTILENUMBER=09 VERTICALTILENUMBER=05 TileID=51009005 NDAYS_COMPOSITED=16 QAPERCENTGOODFPAR=100 QAPERCENTGOODLAI=100 QAPERCENTMAINMETHOD=100 QAPERCENTEMPIRICALMODEL=0 NORTHBOUNDINGCOORDINATE=39.9999999964079 SOUTHBOUNDINGCOORDINATE=29.9999999973059 EASTBOUNDINGCOORDINATE=-92.3664205550513 WESTBOUNDINGCOORDINATE=-117.486656023174 ALGORITHMPACKAGEACCEPTANCEDATE=x-01-2004 ALGORITHMPACKAGEMATURITYCODE=Normal ALGORITHMPACKAGENAME=MCDPR_15A2 ALGORITHMPACKAGEVERSION=v GEOANYABNORMAL=False GEOESTMAXRMSERROR=fifty.0 LONGNAME=MODIS/Terra+Aqua Leafage Surface area Index/FPAR viii-Twenty-four hour period L4 Global 1km SIN Grid PROCESSINGCENTER=MODAPS SYSTEMFILENAME=MYD15A1.A2011192.h09v05.005.2011194222549.hdf, MYD15A1.A2011191.h09v05.005.2011194224921.hdf, MYD15A1.A2011190.h09v05.005.2011192044353.hdf, MYD15A1.A2011189.h09v05.005.2011191043320.hdf, MYD15A1.A2011188.h09v05.005.2011190043751.hdf, MYD15A1.A2011187.h09v05.005.2011189125127.hdf, MYD15A1.A2011186.h09v05.005.2011188044024.hdf, MYD15A1.A2011185.h09v05.005.2011187152544.hdf, MOD15A1.A2011192.h09v05.005.2011194044548.hdf, MOD15A1.A2011191.h09v05.005.2011193042953.hdf, MOD15A1.A2011190.h09v05.005.2011192045720.hdf, MOD15A1.A2011189.h09v05.005.2011191045425.hdf, MOD15A1.A2011188.h09v05.005.2011190045726.hdf, MOD15A1.A2011187.h09v05.005.2011189045204.hdf, MOD15A1.A2011186.h09v05.005.2011188195439.hdf, MOD15A1.A2011185.h09v05.005.2011210211523.hdf NUMBEROFGRANULES=1 GRANULEDAYNIGHTFLAG=Mean solar day GRANULEBEGINNINGDATETIME=2011-07-29T21:xv:23.000Z GRANULEENDINGDATETIME=2011-07-29T21:xv:23.000Z CHARACTERISTICBINANGULARSIZE=30.0 CHARACTERISTICBINSIZE=926.625433055556 DATACOLUMNS=1200 DATAROWS=1200 GLOBALGRIDCOLUMNS=43200 GLOBALGRIDROWS=21600 NADIRDATARESOLUTION=1km MAXIMUMOBSERVATIONS=1 SPSOPARAMETERS=5367, 2680 PROCESSINGENVIRONMENT=Linux minion5559 ii.vi.18-238.12.1.el5PAE #1 SMP Tue May 31 14:02:45 EDT 2011 i686 i686 i386 GNU/Linux DESCRREVISION=5.0 ENGINEERING_DATA= # MOD_PR15A2 (Vers 5.0.4 Rele ten.18.2006 23:59) # MUM API Vers 2.5.viii Rev 104 Rel. 11.fifteen.2000 x:49 (pgs) # (c) 2000 J.K. Burnished, NTSG,LLSD # portions of MUM API past Petr Votava,NTSG Lab,U.Montana # HOST ECS_PGS_VirtualHost Procedure 19540618 # PLATFORM Sys genericunix Vers unknown Release unknown Node (no nodename available) # INIT-TIME Mon Aug 1 eleven:45:xiv 2011 YEARDAY 185 COMPOSITE_PERIOD 24 FIRSTDAY_IN_PERIOD 185 SDS[PGE34_ISG_MBRLUT] %ID 268697600 Rank 2 (664 120) SDS[PGE34_OUTFIELD_PROP] %ID 268697601 Rank ii (78 179) SDS[PGE34_BROWSEFIELD_PROP] %ID 268697602 Rank ii (77 164) READANC SDS[PGE34_ISG_MBRLUT] RANK 2 (664 120) READANC SDS[PGE34_OUTFIELD_PROP] RANK 2 (78 179) READANC SDS[PGE34_BROWSEFIELD_PROP] RANK 2 (77 164) READANC SDS[PGE34_ECSMETA_DICT] RANK 2 (73 110) READANC SDS[PGE34_RELEASE_NOTES] RANK ii (84 98) FLDPROP SDSnam(PGE34_OUTFIELD_PROP)MoleName(PGE34_OUTFIELD_PROP)Status 3 Nelem 13962 BROWSE cvlBrwMol 0 Nelem 57600 Type 21 Status 3 Browse cvlBrwMol 1 Nelem 57600 Type 21 Status iii BROWSE cvlBrwMol 2 Nelem 57600 Type 21 Status three Scan cvlBrwMol 3 Nelem 57600 Blazon 21 Status 3 Browse: NEW GRID ID 4194320 BROWSEFIELD 0 Sum 1725178 Average 29.951 BROWSEFIELD 1 Sum 581056 Average 10.0878 BROWSEFIELD two Sum 358279 Average 6.22012 BROWSEFIELD 3 Sum 8016362 Boilerplate 139.173 BROWSE-DONE: N-Pixels 57600 Invalid offsets: 0 OutOfRange 0 COMPOSITING FPAR FREQUENCIES FPAR 0 34 FPAR one 71 FPAR two 21 FPAR 3 fifty FPAR 4 221 FPAR v 592 FPAR 6 4712 FPAR 7 1445 FPAR 8 522 FPAR nine 1210 FPAR 10 12969 FPAR 11 17195 FPAR 12 83945 FPAR thirteen 197436 FPAR 14 79525 FPAR 15 55842 FPAR sixteen 68815 FPAR 17 40112 FPAR eighteen 62851 FPAR 19 42180 FPAR 20 43324 FPAR 21 34354 FPAR 22 20947 FPAR 23 29184 FPAR 24 35510 FPAR 25 32078 FPAR 26 34318 FPAR 27 24998 FPAR 28 28379 FPAR 29 18195 FPAR 30 17813 FPAR 31 15146 FPAR 32 10109 FPAR 33 14734 FPAR 34 17210 FPAR 35 23089 FPAR 36 14491 FPAR 37 15924 FPAR 38 15249 FPAR 39 15657 FPAR xl 9683 FPAR 41 9847 FPAR 42 7833 FPAR 43 11433 FPAR 44 12093 FPAR 45 10148 FPAR 46 9235 FPAR 47 11742 FPAR 48 10464 FPAR 49 8277 FPAR 50 5914 FPAR 51 8763 FPAR 52 6712 FPAR 53 6807 FPAR 54 9616 FPAR 55 7249 FPAR 56 5771 FPAR 57 6052 FPAR 58 5514 FPAR 59 6665 FPAR 60 5731 FPAR 61 5195 FPAR 62 5484 FPAR 63 4754 FPAR 64 4820 FPAR 65 4963 FPAR 66 3775 FPAR 67 4243 FPAR 68 4047 FPAR 69 3461 FPAR lxx 2613 FPAR 71 3143 FPAR 72 2812 FPAR 73 2463 FPAR 74 1918 FPAR 75 1894 FPAR 76 1938 FPAR 77 1574 FPAR 78 1329 FPAR 79 1287 FPAR 80 1165 FPAR 81 1210 FPAR 82 1268 FPAR 83 1501 FPAR 84 1562 FPAR 85 1466 FPAR 86 1875 FPAR 87 1865 FPAR 88 2695 FPAR 89 8083 FPAR 90 4573 FPAR 91 800 FPAR 92 830 FPAR 93 723 FPAR 94 446 FPAR 95 146 FPAR 96 59 FPAR 97 52 FPAR 98 26 FPAR 99 6 FPAR 100 1 COMPOSITING LAI FREQUENCIES LAI 0 135 LAI i 7017 LAI 2 333047 LAI iii 320752 LAI 4 116468 LAI v 116636 LAI 6 95497 LAI 7 74899 LAI eight 46317 LAI 9 30182 LAI x 38262 LAI xi 28509 LAI 12 20955 LAI 13 16270 LAI 14 18420 LAI xv 13305 LAI 16 13099 LAI 17 10446 LAI eighteen 6287 LAI 19 8236 LAI twenty 6428 LAI 21 5986 LAI 22 5851 LAI 23 4651 LAI 24 3958 LAI 25 4149 LAI 26 3228 LAI 27 2795 LAI 28 2712 LAI 29 2032 LAI 30 1961 LAI 31 1927 LAI 32 1557 LAI 33 1448 LAI 34 1445 LAI 35 1123 LAI 36 1135 LAI 37 991 LAI 38 849 LAI 39 748 LAI 40 771 LAI 41 712 LAI 42 641 LAI 43 655 LAI 44 816 LAI 45 947 LAI 46 965 LAI 47 947 LAI 48 934 LAI 49 948 LAI 50 758 LAI 51 677 LAI 52 662 LAI 53 727 LAI 54 651 LAI 55 643 LAI 56 789 LAI 57 790 LAI 58 613 LAI 59 868 LAI 60 676 LAI 61 776 LAI 62 3217 LAI 63 796 LAI 64 608 LAI 65 2298 LAI 66 4179 LAI 67 166 LAI 68 123 LAI 69 5 COMPOSITING Twenty-four hours Index FREQUENCIES TILE-INDEX 0 Pick Frequency: 177256 TILE-Alphabetize 1 Selection Frequency: 143267 TILE-Index 2 Selection Frequency: 164215 TILE-Index 3 Pick Frequency: 128123 TILE-INDEX 4 Selection Frequency: 168637 TILE-INDEX 5 Pick Frequency: 99292 TILE-INDEX vi Selection Frequency: 110562 TILE-INDEX seven Selection Frequency: 61898 TILE-Alphabetize 8 Selection Frequency: 27366 TILE-Index nine Selection Frequency: 89455 TILE-Index ten Selection Frequency: 32747 TILE-Alphabetize 11 Selection Frequency: 73910 TILE-INDEX 12 Option Frequency: 43557 TILE-Alphabetize 13 Selection Frequency: 26380 TILE-INDEX 14 Selection Frequency: 36942 TILE-Index 15 Selection Frequency: 14464 LOCALGRANULEID [MCD15A2.A2011185.h09v05.005.2011213154534.hdf] ECS QC Percentage North valid (demominator) : 1398071.000000 ECS Total Pixels (denominator) : 1440000 QAPERCENTxx PSA BEFORE CALC QAPERCENTINTERPOLATEDDATA: 0 QAPERCENTMISSINGDATA : 41929 QAPERCENTOUTOFBOUNDSDATA : 41929 QAPERCENTCLOUDCOVER : 334183 QAPERCENTNOTPRODUCEDCLOUD: 0 QAPERCENTNOTPRODUCEDOTHER: 0 QAPERCENTGOODQUALITY : 1391137 QAPERCENTOTHERQUALITY : 48863 QAPERCENTGOODFPAR : 1391137 QAPERCENTGOODLAI : 1391137 QAPERCENTMAINMETHOD : 1391137 QAPERCENTEMPIRICALMODEL : 6934 QAPERCENTNDAYSCOMPOSITED : 16 QAPERCENTTERRA : 1052272 QAPERCENTxx PSA AFTER CALC QAPERCENTINTERPOLATEDDATA: 0 QAPERCENTMISSINGDATA : iii QAPERCENTOUTOFBOUNDSDATA : 3 QAPERCENTCLOUDCOVER : 23 QAPERCENTNOTPRODUCEDCLOUD: 0 QAPERCENTNOTPRODUCEDOTHER: 0 QAPERCENTGOODQUALITY : 100 QAPERCENTOTHERQUALITY : 100 QAPERCENTGOODFPAR : 100 QAPERCENTGOODLAI : 100 QAPERCENTMAINMETHOD : 100 QAPERCENTEMPIRICALMODEL : 0 QAPERCENTNDAYSCOMPOSITED : 16 SESSION ENGINEERING SUMMARY FOR PGE34 8-twenty-four hours FPAR,LAI UM_VERSION U.MONTANA MODIS PGE34 Vers 5.0.4 Rev iv Release 10.eighteen.2006 23:59 Candidate Days: sixteen N. invalid loads : 0 Pixels failing best day : 41929 Pixels ready to make full : 0 Pixels skipped (disqual): 1118070 Unclassified Pixels : 41929 MOD15A1 DAILY Input: /MODAPSops8/annal/f5559/running/AMPM_L10m/6913096/MOD15A1.A2011185.h09v05.005.2011210211523.hdf MOD15A1 DAILY Input: /MODAPSops8/annal/f5559/running/AMPM_L10m/6913096/MOD15A1.A2011186.h09v05.005.2011188195439.hdf MOD15A1 DAILY Input: /MODAPSops8/archive/f5559/running/AMPM_L10m/6913096/MOD15A1.A2011187.h09v05.005.2011189045204.hdf MOD15A1 DAILY Input: /MODAPSops8/archive/f5559/running/AMPM_L10m/6913096/MOD15A1.A2011188.h09v05.005.2011190045726.hdf MOD15A1 DAILY Input: /MODAPSops8/archive/f5559/running/AMPM_L10m/6913096/MOD15A1.A2011189.h09v05.005.2011191045425.hdf MOD15A1 DAILY Input: /MODAPSops8/archive/f5559/running/AMPM_L10m/6913096/MOD15A1.A2011190.h09v05.005.2011192045720.hdf MOD15A1 DAILY Input: /MODAPSops8/annal/f5559/running/AMPM_L10m/6913096/MOD15A1.A2011191.h09v05.005.2011193042953.hdf MOD15A1 DAILY Input: /MODAPSops8/annal/f5559/running/AMPM_L10m/6913096/MOD15A1.A2011192.h09v05.005.2011194044548.hdf MOD15A1 DAILY Input: /MODAPSops8/archive/f5559/running/AMPM_L10m/6913096/MYD15A1.A2011185.h09v05.005.2011187152544.hdf MOD15A1 DAILY Input: /MODAPSops8/archive/f5559/running/AMPM_L10m/6913096/MYD15A1.A2011186.h09v05.005.2011188044024.hdf MOD15A1 DAILY Input: /MODAPSops8/archive/f5559/running/AMPM_L10m/6913096/MYD15A1.A2011187.h09v05.005.2011189125127.hdf MOD15A1 DAILY Input: /MODAPSops8/archive/f5559/running/AMPM_L10m/6913096/MYD15A1.A2011188.h09v05.005.2011190043751.hdf MOD15A1 DAILY Input: /MODAPSops8/annal/f5559/running/AMPM_L10m/6913096/MYD15A1.A2011189.h09v05.005.2011191043320.hdf MOD15A1 DAILY Input: /MODAPSops8/archive/f5559/running/AMPM_L10m/6913096/MYD15A1.A2011190.h09v05.005.2011192044353.hdf MOD15A1 DAILY Input: /MODAPSops8/archive/f5559/running/AMPM_L10m/6913096/MYD15A1.A2011191.h09v05.005.2011194224921.hdf MOD15A1 DAILY Input: /MODAPSops8/archive/f5559/running/AMPM_L10m/6913096/MYD15A1.A2011192.h09v05.005.2011194222549.hdf PGE34 Output : /MODAPSx/archive/f5559/ops8/running/AMPM_L10m/6913096/MCD15A2.1.2011-185T00:00:00.000000Z.51009005.8618434.215230_1.hdf MOD15A2 Ancillary : /MODAPSops8/PGE/AMPM/coeff/PGE34/MCD15A2_ANC_RI4.hdf QAPERCENTxx PSA AFTER CALC QAPERCENTINTERPOLATEDDATA: 0 QAPERCENTMISSINGDATA : 3 QAPERCENTOUTOFBOUNDSDATA : 3 QAPERCENTCLOUDCOVER : 23 QAPERCENTTERRA : 75 QAPERCENTGOODQUALITY : 100 QAPERCENTOTHERQUALITY : 100 QAPERCENTGOODFPAR : 100 QAPERCENTGOODLAI : 100 QAPERCENTMAINMETHOD : 100 QAPERCENTEMPIRICALMODEL : 0 QAPERCENTNDAYSCOMPOSITED : xvi Started Mon Aug one xi:45:14 2011 Ended Mon Aug i 11:45:34 2011 Elapsed Fourth dimension 20 Sec ( 0.33 Min) MOD15A2_FILLVALUE_DOC=MOD15A2 Make full VALUE Legend 255 = _Fillvalue, assigned when: * the MODAGAGG suf. reflectance for channel VIS, NIR was assigned its _Fillvalue, or * land cover pixel itself was assigned _Fillvalus 255 or 254. 254 = state cover assigned equally perennial salt or inland fresh water. 253 = land encompass assigned as arid, thin vegetation (stone, tundra, desert.) 252 = land cover assigned equally perennial snowfall, ice. 251 = state embrace assigned equally "permanent" wetlands/inundated marshlands. 250 = land comprehend assigned as urban/congenital-up. 249 = state cover assigned equally "unclassified" or not able to determine. MOD15A2_FparLai_QC_DOC= FparLai_QC 5 BITFIELDS IN 8 BITWORD MODLAND_QC Outset 0 END 0 VALIDS 2 MODLAND_QC 0 = Practiced Quality (main algorithm with or without saturation) MODLAND_QC ane = Other Quality (back-upwards algorithm or fill value) SENSOR START i Cease 1 VALIDS two SENSOR 0 = Terra SENSOR i = Aqua DEADDETECTOR Commencement 2 END 2 VALIDS ii DEADDETECTOR 0 = Detectors obviously fine for up to 50% of channels 1,2 DEADDETECTOR ane = Dead detectors caused >50% adjacent detector retrieval CLOUDSTATE Outset three Cease 4 VALIDS 4 (this inherited from Aggregate_QC $.25 {0,ane} cloud state) CLOUDSTATE 00 = 0 Pregnant clouds NOT present (articulate) CLOUDSTATE 01 = i Significant clouds WERE nowadays CLOUDSTATE x = 2 Mixed cloud nowadays on pixel CLOUDSTATE eleven = 3 Cloud state non defined,assumed articulate SCF_QC START v End vii VALIDS five SCF_QC 000=0 Main (RT) algorithm used, best result possible (no saturation) SCF_QC 001=1 Chief (RT) algorithm used, saturation occured. Expert, very usable. SCF_QC 010=2 Main algorithm failed due to bad geometry, empirical algorithm used SCF_QC 011=iii Main algorithm faild due to issues other than geometry, empirical algorithm used SCF_QC 100=4 Pixel not produced at all, value coudn't exist retrieved (possible reasons: bad L1B data, unusable MODAGAGG data) MOD15A2_FparExtra_QC_DOC= FparExtra_QC 6 BITFIELDS IN 8 BITWORD LANDSEA PASS-THROUGH Beginning 0 END i VALIDS 4 LANDSEA 00 = 0 State AggrQC(3,5)values{001} LANDSEA 01 = ane SHORE AggrQC(three,five)values{000,010,100} LANDSEA x = ii FRESHWATER AggrQC(iii,5)values{011,101} LANDSEA eleven = 3 OCEAN AggrQC(3,5)values{110,111} SNOW_ICE (from Aggregate_QC bits) Get-go 2 End 2 VALIDS ii SNOW_ICE 0 = No snowfall/water ice detected SNOW_ICE 1 = Snow/ice were detected Droplets Beginning 3 End 3 VALIDS 2 AEROSOL 0 = No or depression atmospheric aerosol levels detected AEROSOL 1 = Average or high aerosol levels detected CIRRUS (from Aggregate_QC bits {eight,ix} ) START 4 END 4 VALIDS 2 CIRRUS 0 = No cirrus detected CIRRUS ane = Cirrus was detected INTERNAL_CLOUD_MASK Commencement five END 5 VALIDS 2 INTERNAL_CLOUD_MASK 0 = No clouds INTERNAL_CLOUD_MASK 1 = Clouds were detected CLOUD_SHADOW Get-go 6 Cease 6 VALIDS 2 CLOUD_SHADOW 0 = No cloud shadow detected CLOUD_SHADOW 1 = Cloud shadow detected SCF_BIOME_MASK Offset vii Cease vii VALIDS 2 SCF_BIOME_MASK 0 = Biome outside interval <1,4> SCF_BIOME_MASK 1 = Biome in interval <i,4> MOD15A2_StdDev_QC_DOC=MOD15A2 STANDARD Divergence FILL VALUE LEGEND 255 = _Fillvalue, assigned when: * the MODAGAGG suf. reflectance for channel VIS, NIR was assigned its _Fillvalue, or * country encompass pixel itself was assigned _Fillvalus 255 or 254. 254 = land encompass assigned equally perennial salt or inland fresh water. 253 = state cover assigned equally arid, thin vegetation (rock, tundra, desert.) 252 = country cover assigned every bit perennial snow, ice. 251 = land cover assigned as "permanent" wetlands/inundated marshlands. 250 = land cover assigned as urban/built-upward. 249 = land cover assigned as "unclassified" or non able to determine. 248 = no standard difference bachelor, pixel produced using fill-in method. MOD15A1_ANC_BUILD_CERT=mtAncUtil v. i.8 Rel. 09.11.2000 17:36 API v. 2.5.half-dozen release 09.14.2000 16:33 Rev.Index 102 (J.Glassy) UM_VERSION=U.MONTANA MODIS PGE34 Vers 5.0.iv Rev 4 Release 10.18.2006 23:59 Subdatasets: SUBDATASET_1_NAME=HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:Fpar_1km SUBDATASET_1_DESC=[1200x1200] Fpar_1km MOD_Grid_MOD15A2 (8-bit unsigned integer) SUBDATASET_2_NAME=HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:Lai_1km SUBDATASET_2_DESC=[1200x1200] Lai_1km MOD_Grid_MOD15A2 (8-bit unsigned integer) SUBDATASET_3_NAME=HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:FparLai_QC SUBDATASET_3_DESC=[1200x1200] FparLai_QC MOD_Grid_MOD15A2 (8-bit unsigned integer) SUBDATASET_4_NAME=HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:FparExtra_QC SUBDATASET_4_DESC=[1200x1200] FparExtra_QC MOD_Grid_MOD15A2 (eight-chip unsigned integer) SUBDATASET_5_NAME=HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:FparStdDev_1km SUBDATASET_5_DESC=[1200x1200] FparStdDev_1km MOD_Grid_MOD15A2 (eight-bit unsigned integer) SUBDATASET_6_NAME=HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:LaiStdDev_1km SUBDATASET_6_DESC=[1200x1200] LaiStdDev_1km MOD_Grid_MOD15A2 (8-bit unsigned integer) Corner Coordinates: Upper Left ( 0.0, 0.0) Lower Left ( 0.0, 512.0) Upper Right ( 512.0, 0.0) Lower Right ( 512.0, 512.0) Middle ( 256.0, 256.0) # Filter lines that do not have BOUNDINGCOORDINATE in them !gdalinfo MCD15A2.A2011185.h09v05.005.2011213154534.hdf | grep BOUNDINGCOORDINATE NORTHBOUNDINGCOORDINATE = 39.9999999964079 SOUTHBOUNDINGCOORDINATE = 29.9999999973059 EASTBOUNDINGCOORDINATE =- 92.3664205550513 WESTBOUNDINGCOORDINATE =- 117.486656023174 We can cheque this confronting e.m. the UNH MODIS tile calculator, just to ostend that we have interpreted the coordinates correctly.
We can apply other beat out GDAL tools, eastward.thou. to perform a reprojection from the native MODIS sinusoidal project, to the Contiguous United States NAD27 Albers Equal Area:
!gdalwarp -of GTiff \ -t_srs '+proj=aea +lat_1=29.5 +lat_2=45.v +lat_0=23 +lon_0=-96 +x_0=0 +y_0=0 +ellps=clrk66 +units=m +no_defs' \ -tr 1000 1000 \ 'HDF4_EOS:EOS_GRID:MCD15A2.A2011185.h09v05.005.2011213154534.hdf:MOD_Grid_MOD15A2:Lai_1km' output_file.tif Creating output file that is 2152P ten 1323L. Processing input file HDF4_EOS:EOS_GRID:MCD15A2.A2011185.h09v05.005.2011213154534.hdf:MOD_Grid_MOD15A2:Lai_1km. Using internal nodata values (eg. 255) for image HDF4_EOS:EOS_GRID:MCD15A2.A2011185.h09v05.005.2011213154534.hdf:MOD_Grid_MOD15A2:Lai_1km. 0...ten...twenty...thirty...40...50...lx...70...80...ninety...100 - done. where MCD15A2.A2011185.h09v05.005.2011213154534.hdf is the name of the input HDF file, MOD_Grid_MOD15A2:Lai_1km is the data product we want, and the rather menacing string +proj=aea +lat_1=29.5 +lat_2=45.5 +lat_0=23 +lon_0=-96 +x_0=0 +y_0=0 +ellps=clrk66 +units=thousand +no_defs specifies the project in Proj4 format. You lot can typically observe the projection you want on spatialreference.org, and just copy and paste the contents of Proj4 definition (remember to surround information technology past quotes). The selection -tr xres yres specifies the desired resolution of the output dataset (1000 by one thousand 1000 in the example above). -of GTiff specifies the GeoTiff format to exist used as as output.
Having some thought what information is in the hdf file and then, we can continue to read the data in inside Python using the GDAL library:
import gdal # Import GDAL library bindings # The file that we shall be using # Needs to exist on current directory filename = 'MCD15A2.A2011185.h09v05.005.2011213154534.hdf' g = gdal . Open ( filename ) # g should at present exist a GDAL dataset, only if the file isn't found # g volition exist none. Allow's examination this: if grand is None : print "Problem opening file %south !" % filename else : impress "File %due south opened fine" % filename subdatasets = g . GetSubDatasets () for fname , name in subdatasets : impress name print " \t " , fname File MCD15A2.A2011185.h09v05.005.2011213154534.hdf opened fine [1200x1200] Fpar_1km MOD_Grid_MOD15A2 (viii-fleck unsigned integer) HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:Fpar_1km [1200x1200] Lai_1km MOD_Grid_MOD15A2 (viii-bit unsigned integer) HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:Lai_1km [1200x1200] FparLai_QC MOD_Grid_MOD15A2 (8-bit unsigned integer) HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:FparLai_QC [1200x1200] FparExtra_QC MOD_Grid_MOD15A2 (8-bit unsigned integer) HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:FparExtra_QC [1200x1200] FparStdDev_1km MOD_Grid_MOD15A2 (8-bit unsigned integer) HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:FparStdDev_1km [1200x1200] LaiStdDev_1km MOD_Grid_MOD15A2 (8-bit unsigned integer) HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:LaiStdDev_1km In the previous code snippet we accept done a number of different things:
- Import the GDAL library
- Open a file with GDAL, storing a handler to the file in g
- Test that g is not None (every bit this indicates failure opening the file. Try changing filename above to something else)
- We then use the GetSubDatasets() method to read out information on the unlike subdatasets available from this file (compare to the output of gdalinfo on the shelf earlier)
- Loop over the retrieved subdatasets to impress the name (human-readable information) and the GDAL filename. This last particular is the filename that you demand to use to tell GDAL to open a item data layer of the half dozen layers present in this example
Let'due south say that nosotros want to access the LAI data. By contrasting the output of the above code (or gdalinfo ) to the contents of the LAI/fAPAR product information page, we notice out that we want the layers for Lai_1km , FparLai_Qc , FparExtra_QC and LaiStdDev_1km .
To read these private datasets, we need to open each of them individually using GDAL, and the GDAL filenames used above:
# Let's create a list with the selected layer names selected_layers = [ "Lai_1km" , "FparLai_QC" , "FparExtra_QC" , "LaiStdDev_1km" ] # We will store the data in a dictionary # Initialise an empty dictionary data = {} # for convenience, we will use string substitution to create a # template for GDAL filenames, which nosotros'll substitute on the wing: file_template = 'HDF4_EOS:EOS_GRID:" %s ":MOD_Grid_MOD15A2: %s ' for i , layer in enumerate ( selected_layers ): this_file = file_template % ( filename , layer ) print "Opening Layer %d : %s " % ( i + 1 , this_file ) m = gdal . Open ( this_file ) if grand is None : raise IOError data [ layer ] = grand . ReadAsArray () print " \t >>> Read %due south !" % layer Opening Layer one: HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:Lai_1km >>> Read Lai_1km! Opening Layer 2: HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:FparLai_QC >>> Read FparLai_QC! Opening Layer 3: HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:FparExtra_QC >>> Read FparExtra_QC! Opening Layer 4: HDF4_EOS:EOS_GRID:"MCD15A2.A2011185.h09v05.005.2011213154534.hdf":MOD_Grid_MOD15A2:LaiStdDev_1km >>> Read LaiStdDev_1km! In the previous code, we take seen a mode of neatly creating the filenames required by GDAL to access the independent datasets: a template string that gets substituted with the filename and the layer name. Note that the presence of double quotes in the template requires us to use single quotes around it. The data is now stored in a dictionary, and can be accessed as due east.g. information['Lai_1km'] :
impress read_data [ 'Lai_1km' ] [[ iii 3 2 ..., 6 8 21] [ 4 3 six ..., 8 18 14] [ 3 12 xi ..., 12 8 8] ..., [ ii 3 ii ..., 18 11 17] [ two 3 iii ..., sixteen 19 15] [ three 2 ii ..., fifteen xvi 15]] Now we have to interpret the LAI values into meaningful quantities. Co-ordinate to the LAI webpage, there is a scale cistron of 0.1 involved for LAI and SD LAI:
lai = data [ 'Lai_1km' ] * 0.1 lai_sd = data [ 'LaiStdDev_1km' ] * 0.ane print "LAI" print lai print "SD" impress lai_sd LAI [[ 0.3 0.three 0.2 ..., 0.6 0.eight 2.1] [ 0.4 0.3 0.vi ..., 0.8 one.viii one.4] [ 0.3 1.2 1.one ..., one.2 0.8 0.8] ..., [ 0.ii 0.3 0.two ..., ane.8 1.1 1.7] [ 0.2 0.3 0.3 ..., 1.six 1.ix 1.five] [ 0.3 0.2 0.ii ..., 1.5 1.6 1.five]] SD [[ 0.two 0.2 0.1 ..., 0.2 0.1 0.3] [ 0.ii 0.ii 0.two ..., 0.2 0.3 0.2] [ 0. 0.one 0.2 ..., 0.1 0.ii 0.2] ..., [ 0.1 0.1 0.1 ..., 0.iii 0. 0.ane] [ 0.ane 0.1 0.one ..., 0.2 0.2 0.i] [ 0.ane 0.1 0.1 ..., 0.1 0.2 0.one]] Some SD values are clearly given as 0.0, which is unlikely to be true. We should so examine the QC (Quality Control) information. The codes for this are also given on the LAI product folio. They are described equally fleck combinations:
| Bit No. | Parameter Name | Scrap Combination | Explanation |
|---|---|---|---|
| 0 | MODLAND_QC $.25 | 0 | Good quality (main algorithm with or without saturation) |
| 1 | Other Quality (back-up algorithm or fill up values) | ||
| one | Sensor | 0 | TERRA |
| i | AQUA | ||
| ii | DeadDetector | 0 | Detectors apparently fine for up to 50% of channels ane two |
| ane | Expressionless detectors caused >l% adjacent detector retrieval | ||
| 3-4 | CloudState | 00 | Significant clouds NOT nowadays (clear) |
| 01 | Significant clouds WERE present | ||
| 10 | Mixed clouds nowadays on pixel | ||
| 11 | Cloud state not defined assumed articulate | ||
| 5-7 | CF_QC | 000 | Main (RT) method used all-time effect possible (no saturation) |
| 001 | Master (RT) method used with saturation. Skilful very usable | ||
| 010 | Principal (RT) method failed due to bad geometry empirical algorithm used | ||
| 011 | Chief (RT) method failed due to problems other than geometry empirical algorithm used | ||
| 100 | Pixel non produced at all value coudn't exist retrieved (possible reasons: bad L1B information unusable MODAGAGG data) |
In using this information, it is upwards to the use which data he/she wants to laissez passer through for any farther processing. There are clearly merchandise-offs: if you wait for only the highest quality data, then the number of samples is probable to be lower than if you were more tolerant. But if you are too tolerant, you will get spurious results. You may find useful information on how to convert from actual QA flags to diagnostics in this page (they focus on NDVI/EVI, simply the theory is the aforementioned).
Only let's just say that we want to utilize simply the highest quality information. From the table higher up, these data are given by the CF_QC flag being set to 000 or 001 , or in other words 00000000 and 00000001 , or 0 and 1 in decimal:
import numpy as np import pylab as plt qc = data [ 'FparLai_QC' ] # Get the QC data # Create a mask like the LAI production mask = np . zeros_like ( lai ) . astype ( bool ) # Fill the mask where the conditions are met mask = np . where ( qc == 0 , True , False ) mask = np . where ( qc == 1 , True , mask ) # Select a blackness/white colormap cmap = plt . cm . gray # plot the data plt . imshow ( mask , interpolation = 'nearest' , cmap = cmap ) # add together a colorbar plt . colorbar ( ticks = [ 0 , 1 ]) <matplotlib.colorbar.Colorbar instance at 0xbb01b00> 
To plot LAI only for the valid pixels ( mask == True ). We can used masked arrays for this. Masked arrays are like normal arrays, merely they have an associated mask, which in this case is mask define above. Nosotros shall also cull another colormap (at that place are lots to cull from), and set values outside the 0.1 and four to be shown every bit blackness pixels.
cmap = plt . cm . Greens cmap . set_bad ( 'one thousand' ) laim = np . ma . array ( lai , mask = mask ) plt . imshow ( laim , cmap = cmap , interpolation = 'nearest' , vmin = 0.ane , vmax = 4 ) plt . colorbar () <matplotlib.colorbar.Colorbar case at 0xd429e18> 
Similarly, nosotros can practise a similar thing for Standard Deviation
cmap = plt . cm . spectral cmap . set_bad ( 'k' ) stdm = np . ma . array ( lai_sd , mask = mask ) plt . imshow ( stdm , cmap = cmap , interpolation = 'nearest' , vmin = 0.001 , vmax = 0.five ) plt . colorbar () <matplotlib.colorbar.Colorbar case at 0x2b8ee20374d0> 
Exercise 1¶
For the moment, nosotros will suppose this information masking to be sufficient. Nevertheless, closer inspection of the production data folio would show us that the data can take on various Fill up Values which any data user should check for.
Modify the code nosotros have developed higher up to also cheque that the data are non unwanted 'fill values' and use this to modify the data mask.
Hint Annotation that these make full values are practical to Lai_1km, not the QC data, so you take to check values in that dataset and modify the mask accordingly.
Practice two¶
We also accept access to FparExtra_QC
qc1 = data [ 'FparExtra_QC' ] The data values in FparExtra_QC are:
| Flake No. | Parameter Name | Bit Combination | Explanation |
|---|---|---|---|
| 0-1 | LandSea | 00 | State |
| 01 | Shore | ||
| 10 | Freshwater | ||
| 11 | Ocean | ||
| 2 | Snow/Ice | 0 | No snow/water ice detected |
| one | Snow/water ice detected | ||
| 3 | Aerosol | 0 | No or low atmospheric aerosol levels detected |
| one | Average or high atmospheric droplets levels detected | ||
| iv | Cirrus | 0 | No cirrus detected |
| 1 | Cirrus detected | ||
| 5 | Cloud | 0 | No deject detected |
| 1 | Deject detected | ||
| half dozen | Cloud shadow | 0 | No cloud shadow detected |
| 1 | Cloud shadowdetected | ||
| seven | Biome_mask | 0 | Biome outside interval i-4 |
| 1 | Biome in interval one-iv |
Use this QC dataset to make certain that but Land pixels are passed in the mask, and apply other data quality measures as appropriate.
Hint this is much the same as the exercise above, only looking at a different QC dataset. The key to doing this is to place the fleck codes for combinations that you lot want to ready or unset.
You tin can find an example of how to exercise this here[DEADLINK]. This filtering shouldn't make much difference in this case, as the tile is mostly land pixels. Still, consider tile h17v03 (role of the UK) for the calendar month of June (e.one thousand. MCD15A2.A2011185.h17v03.005.2011213154608.hdf )
TODO equally NASA website is down!!!!!
UK LAI map
shell-init: fault retrieving current directory: getcwd: cannot access parent directories: No such file or directory pwd: mistake retrieving electric current directory: getcwd: cannot access parent directories: No such file or directory pwd: fault retrieving current directory: getcwd: cannot admission parent directories: No such file or directory def css_styling (): from IPython.display import brandish , HTML styles = "https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/raw/master/styles/custom.css" return HTML ( styles ) css_styling () Source: http://www2.geog.ucl.ac.uk/~plewis/geogg122_current/_build/html/ChapterX_GDAL/GDAL_HDF.html
0 Response to "How to Read Hdf in Python by Certain Incidies"
Post a Comment